Function: Advanced Document Controlled Expiry

Enterprise Edition

Goal: Purge a document automatically based on self-contained start and duration fields.

This function advancedDocControlledSelfExpiry demonstrates self-expiry of a document; for example, a user trial.
Requires Eventing Storage (or metadata collection) and a "source" collection.
Needs a Binding of type "bucket alias" (as documented in the Scriptlet).
When documents are created, they will have no expiration value. This function processes the initial mutation to calculate and set the proper TTL.
In Couchbase, when using a simple integer expiry value (as opposed to a proper date or time object), the expiration can be specified in two ways:
- As an offset from the current time. If the absolute value of the expiry is less than 30 days (60 * 60 * 24 * 30 seconds), it is considered an offset.
- As an absolute Unix time stamp. If the value is greater than 30 days (60 * 60 * 24 * 30 seconds), it is considered an absolute time stamp.
- As described in Expiration, if a "Bucket Max Time-To-Live" is set (specified in seconds), it is an enforced hard upper limit. As such, any subsequent document mutation (by SQL++, Eventing, or any Couchbase SDK) will result in the document having its expiration adjusted and set to the bucket’s maximum TTL if the operation has:
  - No TTL.
  - A TTL of zero.
  - A TTL greater than the bucket TTL.
As we are using Advanced Bucket Accessors setting document expirations (or TTLs) we use a JavaScript Data object.
Will operate on any document with type == "trial_customers".
Will ignore any doc with a non-zero TTL
This is different than setting a TTL on a bucket or a collection which will typically update (or extend) the TTL of a document on each mutation.

advancedDocControlledSelfExpiry
Input Data/Mutation
Output Data/Mutation

Two variants of this function are available - a 6.6 version that relies on SQL++ and a 6.6.1+/7.0.0+ version (this Function) that directly sets the expiration. You can completely avoid N1QL(…) and use couchbase.replace(bucket_binding, meta, doc) as the advancedDocControlledSelfExpiry variant is much faster.

advancedDocControlledSelfExpiry (direct TTL)

// To run configure the settings for this Function, advancedDocControlledSelfExpiry, as follows:
//
// Version 7.1+
//   "Function Scope"
//     *.* (or try bulk.data if non-privileged)
// Version 7.0+
//   "Listen to Location"
//     bulk.data.source
//   "Eventing Storage"
//     rr100.eventing.metadata
//   Binding(s)
//    1. "binding type", "alias name...", "bucket.scope.collection", "Access"
//       "bucket alias", "src_col",       "bulk.data.source",        "read and write"
//
// Version 6.6.1
//   "Source Bucket"
//     source
//   "MetaData Bucket"
//     metadata
//   Binding(s)
//    1. "binding type", "alias name...", "bucket", "Access"
//       "bucket alias", "src_col",       "source", "read and write"

function OnUpdate(doc, meta) {
    // Filter items that don't have been updated
    if (meta.expiration !== 0) {
        log(meta.id, "IGNORE expiration "+meta.expiration+" !== 0 or "+
            new Date(meta.expiration).toString());
        return;
    }

    // Optional filter to a specic field like 'type'
    if (doc.type !== 'trial_customers') return;

    // Our expiry is based on a JavaScript date parsable field, it must exist
    if (!doc.trialStartDate || !doc.trialDurationDays) return;

    // Convert the doc's field timeStamp and convert to unix epoch time (in ms.).
    var docTimeStampMs = Date.parse(doc.trialStartDate);

    var keepDocForMs = doc.trialDurationDays * 1000 * 60 * 60 * 24 ;
    var nowMs = Date.now();  // get current unix time (in ms.).

    // Archive if we have kept it for too long no need to set the expiration
    if( nowMs >= (docTimeStampMs + keepDocForMs) ) {

        // Delete the document form the source collection via the map alias
        delete src_col[meta.id];

        log(meta.id, "DELETE from src_col to dst_bkt alias as our expiration " +
            new Date(docTimeStampMs + keepDocForMs).toString()) + " is already past";
    } else {
        var key = meta.id;
        //set the meta.expiration=ttlMs
        var	ttlMs = docTimeStampMs + keepDocForMs;

        if (ttlMs !== 0) {
            log(meta.id, "UPDATE expiration "+meta.expiration+" === 0 set to "+
                ttlMs+" or " + new Date(ttlMs).toString());
            // Advanced Bucket Accessors use JavaScript Date objects
            var expiryDate = new Date(ttlMs);
            // This is 4X to 5X faster than using N1QL(...) and no need to worry about recursion.
            var res = couchbase.replace(src_col,{"id":meta.id,"expiry_date":expiryDate},doc);
            if (!res.success) {
                log(meta.id,'Setting TTL to',expiryDate,'failed',res);
            }
        }
    }
}

We want to create a test set of four (4) documents, use the Query Editor to insert the the data items (you do not need an index).

Note, if the today is past 08-25-2021 (MM-DD-YYYY) just change the trialStartDate for the last two records to at least 90 days from now.

  UPSERT INTO `bulk`.`data`.`source` (KEY,VALUE)
  VALUES ( "trial_customers::0", {
    "type": "trial_customers",
    "id": 0,
    "trialStartDate": "08-25-2019",
    "trialDurationDays": 30,
    "note": "this is old will get immeadiately deleted"
  } ),
  VALUES ( "trial_customers::1",
  {
    "type": "trial_customers",
    "id": 1,
    "trialStartDate": "01-27-2020",
    "trialDurationDays": 30,
    "note": "this is old will get immeadiately deleted"
  } ),
  VALUES ( "trial_customers::2",
  {
    "type": "trial_customers",
    "id": 2,
    "trialStartDate": "08-25-2021",
    "trialDurationDays": 30,
    "note": "this will get an exiration set"
  } ),
  VALUES ( "trial_customers::3",
  {
    "type": "trial_customers",
    "id": 3,
    "trialStartDate": "08-26-2021",
    "trialDurationDays": 60,
    "note": "this will get an exiration set"
  } );

NEW/OUTPUT: KEY trial_customers::2

{
  "id": 2,
  "note": "this will get an exiration set",
  "trialDurationDays": 30,
  "trialStartDate": "08-25-2021",
  "type": "trial_customers"
}

NEW/OUTPUT: KEY trial_customers::3

{
  "id": 3,
  "note": "this will get an exiration set",
  "trialDurationDays": 60,
  "trialStartDate": "08-26-2021",
  "type": "trial_customers"
}

We end up with two (2) of the four documents (obviously you may need to adjust the {sqlpp} INSERT in a few months as all the document would be immediately deleted).

* "trial_customers::0" was deleted
* "trial_customers::1" was deleted
* "trial_customers::2" has an meta.expiration set for 1632466800 (or 2021-09-24 07:00:00 UTC) in it's metadata
* "trial_customers::3" has an meta.expiration set for 1635145200 (or 2021-10-25 07:00:00 UTC) in it's metadata