Delete v Expiry
Goal: Differentiate between a deletion or an expiration
Implementation:
Create a JavaScript Function that contains an OnDelete handler, which runs whenever a document is deleted or expired (or mutated). The Eventing Function merely logs whether the document was deleted or was expired from a collection. This function accesses the 'expired' field of the second optional argument to the handler.
Preparations:
For this example, two (2) buckets 'bulk' and 'rr100' are required where the latter is intended to be 100% resident. Create the buckets with a minimum size of 100MB. For information on buckets, see Create a Bucket. Within the buckets we need two (2) keyspaces 'bulk.data.source' and 'rr100.eventing.metadata' (we loosely follow this organization).
For the Function Scope or RBAC grouping, we will use the 'bulk.data', assuming you have the role of either "Full Admin" or "Eventing Full Admin". For standard or non-privileged users refer to Eventing Role-Based Access Control.
If you run a version of Couchbase prior to 7.0 you can just create the buckets 'source' and 'metadata' and run this example. Furthermore if your cluster was subsequently upgraded from say 6.6.2 to 7.0 your data would be moved to 'source._default._default' and 'metadata._default._default' and your Eventing Function would be seamlessly upgraded to use the new keyspaces and continue to run correctly.
For complete details on how to set up your keyspaces refer to creating buckets and creating scopes and collections.
The Eventing Storage keyspace, in this case 'rr100.eventing.metadata', is for the sole use of the Eventing system, do not add, modify, or delete documents from it. In addition do not drop or flush or delete the containing bucket (or delete this collection) while you have any deployed Eventing functions. In a single tenancy deployment this collection can be shared with other Eventing functions. |
You will need to run cbc (the command-line KV client) or alternatively an SDK python script or Java program to create or update a document in the 'source' bucket, with an expiration time of 600 seconds.
Procedure:
-
The example requires a document to be created in the 'source' collection with a key of SampleDocument2, a value of {'a_key': 'a_value'}, and most importantly that the document’s expiration (or TTL) set to 600 seconds or 10 minutes).
There are several methods to make a test document with an expiration set. The easiest is most likely using SQL++. However you can use cbc or any Couchbase SDK (the command-line KV client is compiled from the C SDK). For example you can use a Python script or a complied Java program.
- Using the Query Workbench
UPSERT INTO `bulk`.`data`.`source` (KEY, VALUE) VALUES ("SampleDocument2", {"a_key":"a_value"}, {"expiration":600});
Issue the above command in the Query Workbench of the UI.
For information on setting document expiration times via SQL++, refer to Insert a document with expiration
- On Linux
/opt/couchbase/bin/cbc \ create SampleDocument2 -V '{"a_key": "a_value"}' \ -U couchbase://localhost/source \ --scope=data --collection=source \ -u Administrator -P password \ --expiry=600
on macOS
/Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbc \ create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \ --scope=data --collection=source \ -u Administrator -P password \ --expiry=600
on Windows
"C:\Program Files\Couchbase\Server\bin\cbc" ^ create SampleDocument2 -V "{'a_key': 'a_value'}" -U couchbase://localhost/source ^ --scope=data --collection=source ^ -u Administrator -P password ^ --expiry=600
Use the command-line KV client, e.g. the cbc binary, and cut-n-paste one of the above commands to create the needed sample document.
We are passing --expiry the number of time in seconds from now at which the item should expire. However if you want an expiry over 30 days you must use the number of seconds since Unix Epoch.
On macOS (or OS-X) if you get a 'dyld: Library not loaded' when running cbc a solution is documented in MB-37768.
For information on the cbc tool, refer to Using the command-line KV client.
#!/usr/bin/python3 import sys import couchbase.collection import couchbase.subdocument as SD from couchbase.cluster import Cluster, ClusterOptions from couchbase_core.cluster import PasswordAuthenticator from couchbase.durability import ServerDurability, Durability from datetime import timedelta pa = PasswordAuthenticator('Administrator', 'password') cluster = Cluster('couchbase://127.0.0.1', ClusterOptions(pa)) bucket = cluster.bucket('bulk') collection = bucket.scope('data').collection('source') try: document = dict( a_key="a_value" ) result = collection.upsert( 'SampleDocument2', document, expiry=timedelta(minutes=10) ) print("UPSERT SUCCESS") print("cas result:", result.cas) except: print("exception:", sys.exc_info())
Make an executable script as above and then it. Alternatively run the command python3 to start a Python session then cut-n-paste the above line (without the
#!/usr/bin/python3
line) to create the needed sample document and then ^D (or ctrl-D) to close the Python session.For information on the Couchbase Python SDK, refer to Start Using the Python SDK.
A Java 3.0 SDK program example
// Must use the Collections API package com.jonstrabala; import java.time.Duration; import com.couchbase.client.java.*; import com.couchbase.client.java.json.JsonObject; import static com.couchbase.client.java.kv.UpsertOptions.upsertOptions; public class DocExpiryTestCC { public static void main(String... args) throws Exception { // Note, if not on the server you need to change "localhost" to your DNS name or IP Cluster cluster = Cluster.connect("localhost", "Administrator", "password"); Bucket bucket = cluster.bucket("bulk"); // Collection collection = bucket.defaultCollection(); Collection collection = bucket.scope("data").collection("source"); String docID = "SampleDocument2"; Duration dura = Duration.ofMinutes(10); try { collection.upsert( docID, JsonObject.create().put("a_key", "a_value"), upsertOptions().expiry(dura) ); System.out.println("docID: " + docID + " expires in " + dura.getSeconds()); } catch (Exception e) { System.out.println("upsert error for docID: " + docID + " " + e); } bucket = null; collection = null; cluster.disconnect(Duration.ofSeconds(2000)); } }
Download the proper SDK and then compile and run one of the above Java programs
For information on the Couchbase Java SDK, refer to Start Using the Java SDK.
-
You now have a document in collection 'source' (keyspace
bulk
.data
.source
) with an expiration set. -
To verify that your new document was created, access the Couchbase Web Console > Documents page and click the Documents then select the keyspace
bulk
.data
.source
. The new document gets displayed automatically (as this page will attempt to list the first few items). You will see one (1) document in thebulk
.data
.source
keyspace (this will disappear on the document’s expiry of 10 minutes). -
[Optional Step] Click on the document’s id, SampleDocument2 to view the documents Data and also the documents Metadata information. Note that the "expiration" field in the Metadata is non-zero (set to a Unix timestamp in seconds since epoch).
-
From the Couchbase Web Console > Eventing page, click ADD FUNCTION, to add a new Function. The ADD FUNCTION dialog appears.
-
In the ADD FUNCTION dialog, for individual Function elements provide the below information:
-
For the Function Scope drop-down, select 'bulk.data' as the RBAC grouping.
-
For the Listen To Location drop-down, select bulk, data, source as the keyspace.
-
For the Eventing Storage drop-down, select rr100, eventing, metadata as the keyspace.
-
Enter delete_v_expiry as the name of the Function you are creating in the Function Name text-box.
-
Leave the "Deployment Feed Boundary" as Everything.
-
[Optional Step] Enter text Function to show how to determine if a document removal is an expiration or a deletion, in the Description text-box.
-
For the Settings option, use the default values.
-
Skip the Bindings options
-
After configuring your settings the ADD FUNCTION dialog should look like this:
-
-
After providing all the required information in the ADD FUNCTION dialog, click Next: Add Code. The delete_v_expiry dialog appears.
-
The delete_v_expiry dialog initially contains a placeholder code block. You will substitute your actual delete_v_expiry code in this block.
-
Copy the following Function, and paste it in the placeholder code block of delete_v_expiry dialog.
function OnDelete(meta, options) { if (options.expired) { log("doc expired:",meta.id); } else { log("doc deleted:",meta.id); } }
After pasting, the screen appears as displayed below:
-
Click Save and Return.
-
-
From the Eventing screen, click the delete_v_expiry function to select it, then click Deploy.
-
Click Deploy Function.
-
-
The Eventing function is deployed and starts running within a few seconds. From this point, the defined Function is executed on all existing documents and on subsequent mutations.
-
When its expiration time is reached, an item is deleted as soon as one of the following occurs:
-
An attempt is made to access the item.
-
The expiry pager is run (default every 60 minutes).
-
Compaction is run.
-
-
Therefore we first wait the full 10 minute period from the creation of SampleDocument2 and then we will try to access the document to expedite the expiry occurrence.
-
Access the Couchbase Web Console > Buckets page.
-
You may see document count of one or zero depending in the Bucket bulk if the expiry pager has run.
-
-
Click on the Documents in the UI you will see one (1) document in the
bulk
.data
.source
keyspace (this will disappear on the document’s expiry of 10 minutes). If the expiration on the document SampleDocument2 has occurred there will be no documents found. -
Access the Couchbase Web Console > Buckets page for a second time.
-
You should see the document count is zero for the bucket source as we have attempted to access the item and it has been recognized and marked as an expired tombstone.
-
-
Access the Couchbase Web Console > Eventing page, click the function name delete_v_expiry.
-
You should see the following statistics under the Deployment Statistics:
-
-
Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).
-
You should see that the document has expired.
2022-05-10T15:07:46.442-07:00 [INFO] "doc expired:" "SampleDocument2"
-
-
Now let’s create another document and perform a normal delete on it. Access the Couchbase Web Console > Buckets page and click the Scopes and Collections link of the bulk bucket.
-
Click Documents in the upper right banner for the data scope.
-
Select the keyspace bulk, data, source
-
You should see no user records.
-
Click Add Document in the upper right banner
-
For the ID in the Create New Document dialog specify SampleDocument
ID [ SampleDocument3 ]
-
For the document body in the Create New Document dialog, the following text is displayed:
{ "click": "to edit", "with JSON": "there are no reserved field names" }
-
just keep the above text
-
Click Save.
-
-
Now click the trash can icon to delete the SampleDocument3 you just created. Select Continue to confirm the deletion.
-
Access the Couchbase Web Console > Eventing page, click the function name delete_v_expiry.
-
You should see following statistics under the Deployment Statistics:
-
-
Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).
-
Here we see the document was deleted.
2022-05-10T15:10:18.565-07:00 [INFO] "doc deleted:" "SampleDocument3"
-
Cleanup:
Go to the Eventing portion of the UI and undeploy the Function delete_v_expiry, this will remove the 1280 documents for each function from the 'rr100.eventing.metadata' collection (in the Bucket view of the UI). Remember you may only delete the 'rr100.eventing.metadata' keyspace if there are no deployed Eventing Functions.
Now flush the 'bulk' bucket if you plan to run other examples (you may need to Edit the bucket 'bulk' and enable the flush capability).