DynamoDB backend: support for time to live field

[philvarner]

In FilmDrop UI's use of Element84/titiler-mosaicjson, we use the mosaic in an ephemeral way, such that a mosaic is discarded as soon as the user changes the search or moves away. Currently, the DynamoDB backend keeps all of these entries that are never going to be used again.

I'd like to add support in the DynamoDB backend to support passing an optional timeToLive parameter that would allow DynamoDB to delete these entries after the TTL. I can do this work, but wanted buy-in on the approach. The simplest option would be to add kwargs to BaseBackend#write, and the backends can do whatever they want with those. Another option would be to add an argument like additionalValues of type Dict[str, Any] and these values would be written as additional fields in the DynamoDB entry, so I could configure my Table to use a TTL field of timeToLive, and then pass a dict like { "timeToLive": 1696430626} that would get written with each DDB Item.

[vincentsarago]

@philvarner I think what you want is just to create a custom write method which accept a ttl

cogeo-mosaic/cogeo_mosaic/backends/dynamodb.py

Lines 116 to 155 in 4f88749

	def write(self, overwrite: bool = False, **kwargs: Any):
	"""Write mosaicjson document to AWS DynamoDB.
	Args:
	overwrite (bool): delete old mosaic items inthe Table.
	**kwargs (any): Options forwarded to `dynamodb.create_table`
	Raises:
	MosaicExistsError: If mosaic already exists in the Table.
	"""
	if not self._table_exists():
	self._create_table(**kwargs)
	if self._mosaic_exists():
	if not overwrite:
	raise MosaicExistsError(
	f"Mosaic already exists in {self.table_name}, use `overwrite=True`."
	)
	self.delete()
	items: List[Dict[str, Any]] = []
	# Create Metadata item
	# Note: `parse_float=Decimal` is required because DynamoDB requires all numbers to be
	# in Decimal type (ref: https://blog.ruanbekker.com/blog/2019/02/05/convert-float-to-decimal-data-types-for-boto3-dynamodb-using-python/)
	meta = json.loads(
	self.mosaic_def.model_dump_json(exclude={"tiles"}), parse_float=Decimal
	)
	items.append(
	{"quadkey": self._metadata_quadkey, "mosaicId": self.mosaic_name, **meta}
	)
	# Create Tile items
	for quadkey, assets in self.mosaic_def.tiles.items():
	items.append(
	{"mosaicId": self.mosaic_name, "quadkey": quadkey, "assets": assets}
	)
	self._write_items(items)

such as

    def write(self, overwrite: bool = False, ttl: Optional[int] = None, **kwargs: Any):
        if not self._table_exists():
            self._create_table(**kwargs)

        if self._mosaic_exists():
            if not overwrite:
                raise MosaicExistsError(
                    f"Mosaic already exists in {self.table_name}, use `overwrite=True`."
                )
            self.delete()

        items: List[Dict[str, Any]] = []

        # Create Metadata item
        # Note: `parse_float=Decimal` is required because DynamoDB requires all numbers to be
        # in Decimal type (ref: https://blog.ruanbekker.com/blog/2019/02/05/convert-float-to-decimal-data-types-for-boto3-dynamodb-using-python/)
        meta = json.loads(
            self.mosaic_def.model_dump_json(exclude={"tiles"}), parse_float=Decimal
        )
        if ttl:
            meta["ttl"] = ttl
        items.append(
            {"quadkey": self._metadata_quadkey, "mosaicId": self.mosaic_name, **meta}
        )

        # Create Tile items
        for quadkey, assets in self.mosaic_def.tiles.items():
            item = {"mosaicId": self.mosaic_name, "quadkey": quadkey, "assets": assets}
            if ttl:
                item["ttl"] = ttl
            items.append(item )

        self._write_items(items)

DynamoDB TTL seems something that could be useful useful to more users (🤷) so we could add it by default in this repo

[philvarner]

Thanks for the feedback! I'll reply again if we actually do work on this.