future3 - Feature "sync shared" (#2009)

* first callback test

* default sync_shared settings added

* moved test callback methods

* fixed logger call

* changed logger name

* test rpc call sync_folder

* settings added

* added test rsync call (subprocess)

* fixed path

* fixed paths

* fixed errorlogging

* changed subprocess shell=false

* added player update

* fixed subprocess args

* fixed rsync parameter

* update database on caller. added return value

* added "wait for database update"

* added server and directory checks
loglevels adjusted

* fixed sync for subfolder shortcuts

* fixed ignored files

* refactored path handling
use os.path instead of string concatenation
fixed handling for abs path folder name

* refactored logic in control class

* added check for "on_rfid_scan_enabled"

* sync_full added

* added check for feature activation

* correction of bool value handling
evaluate to false if settings not correctly set (e.g. as string)

* fix flake8 errors

* update log message and fix result code

* added ssh support

* refactored paths for run_params

* speed up ssh mode
perform less checks for folder existence

* added sync_change_on_rfid_scan

* updated default settings
format like ConfigHandler would save it

* added command binding for Ui

* fixed binding of command options
and made them lowercase

* changed invalid parameter handling

* added sync_card_database

update card id only on rfid scan sync
overwrite on full sync

* refactorings

added locking on cfg access
updated methodnames
updated logging
methods reordered
flake8 corrections

* exclude folder.conf if existing from V2.x

* changed call on rfid scan to callback

* fix flake8 errors

* fix indendation for JS

* combine settings of credentials for modes

* naming convention

* refactored function names to be more clear

* changed options of sync_change_on_rfid_scan

options changed from  "true"/"false" to "enable"/"disable"

* moved identical prechecks to functions

* renamed "sync_full" to "sync_all"

* Fix function calls

fix for: moved identical prechecks to functions

* renamed "sync_full" to "sync_all"

correction for logging

* added "update_wait" and fixed to much locking

* changed call on play_card to callback

* changed precheck names to "is sync enabled"

* updated function names "is_file" and "is_dir"

* reduced nesting complexity

* Changed rfid callback state to Enum

renamed callback class
add state as enum

* Changed playcontent callback state to Enum

added state as num
moved callback and enum to seperate class
callback class with generic to be able to use in more play functions

* fix import

* fixed generic type definition

* harmonised precheck for sync_change_on_rfid_scan

* refactored methods to util class

* renamed syncutil to syncutils. fixed import

* fixed flake8

* Moved syncutils up

* renamed module sync_shared to rfidcards

* renamed sync_shared to sync_rfidcards

* fix flake8

* updated documentation

* Updated translation for en

* Updated language

* Updated language

* Update docs

---------

Co-authored-by: pabera <[email protected]>

docs/sphinx/index.rst

@@ -34,6 +34,7 @@ Welcome to RPi Jukebox RFID's documentation!
    userguide/rpc_command_alias_reference
    userguide/carddatabase
    userguide/autohotspot
+   userguide/sync_rfidcards
 
 
 .. toctree::

docs/sphinx/userguide/sync_rfidcards.rst

@@ -0,0 +1,71 @@
+Syncronisation RFID Cards
+*************************
+
+This component handles the synchronisation of RFID cards (audiofolder and card database entries).
+
+It allows to manage card database entries and audiofiles of one to many Phonieboxes
+in a central place (e.g. NAS, primary Phoniebox etc.) in the network,
+but allows to play the audio offline once the data has synced.
+The synchronisation can be initiated with the command ``sync-all``
+and optionally on every RFID scan for a particular CardID and its corresponding audiofolder.
+To execute the ``sync-all`` command, bind a RFID card to the command.
+For the "RFID scan sync" feature, activate the option in the configuration
+or bind a RFID card to the command for dynamic activation or deactivation.
+
+Synchronisation
+---------------
+
+The synchronisation will be FROM a server TO the Phoniebox, overriding existing files.
+A local configuration will be lost after the synchronization.
+If you want to make the initial setup e.g. via WebUi copy the files and use it as a base for the server.
+
+To access the files on the server, 2 modes are supported: SSH or MOUNT.
+Please make sure you have the correct access rights to the source and use key-based authentication for SSH.
+
+RFID scan sync
+^^^^^^^^^^^^^^
+If the feature "RFID scan sync" is activated, there will be a check on every RFID scan against the server
+if a matching card entry and audiofolder is available. If so, changes will be synced.
+The playback will be delayed for the time the data is transfered (see "sync-all" to use a full synchronization if a lot of new files have been added).
+If the server is not reachable, the check will be aborted after the timeout.
+Therfore, an unreachable server will cause a delay (see commands to toggle activation state).
+Deleted card entries / audiofolders (not the contained items) will not be purged locally if deleted on remote.
+This is also true for changed card entries (the old audiofolder / -files will remain). To remove not existing items us a "sync-all".
+
+Configuration
+-------------
+
+Set the corresponding setting in ``shared\settings\jukebox.yaml`` to activate this feature.
+
+.. code-block:: yaml
+
+    modules:
+        named:
+            ...
+            sync_rfidcards: synchronisation.rfidcards
+
+    ...
+    sync_rfidcards:
+        enable: false
+        config_file: ../../shared/settings/sync_rfidcards.yaml
+
+The settings file (``shared\settings\sync_rfidcards.yaml``) contains the following configuration
+
+.. code-block:: yaml
+
+    sync_rfidcards:
+        # Holds the activation state of the optional feature "RFID scan sync". Values are "TRUE" or "FALSE"
+        on_rfid_scan_enabled: true # bool
+        # Server Access mode. MOUNT or SSH
+        mode: mount # 'mount' or 'ssh'
+        credentials:
+            # IP or hostname of the server (used to check connectivity and for SSH mode). e.g. "192.168.0.2" or "myhomeserver.local"
+            server: ''
+            # Port (used to check connectivity and for SSH mode). e.g. "80" or "22"
+            port:  # int
+            # Timeout to reach the server (in seconds) (used to check connectivity). e.g. 1
+            timeout: 1 # int
+            # Path to the shared files to sync (without trailing slash) (remote path for SSH mode or local path for MOUNT mode). e.g. "/mnt/Phoniebox"
+            path: ''
+            # Username if SSH mode is used.
+            username: ''

resources/default-settings/jukebox.default.yaml

@@ -18,6 +18,7 @@ modules:
     host: hostif.linux
     bluetooth_audio_buttons: controls.bluetooth_audio_buttons
     gpio: gpio.gpioz.plugin
+    sync_rfidcards: synchronisation.rfidcards
   others:
   - music_cover_art
   - misc
@@ -142,3 +143,6 @@ speaking_text:
   speak_punct: False
   # Must be one of: female, male, croak, whisper
   voice: female
+sync_rfidcards:
+  enable: false
+  config_file: ../../shared/settings/sync_rfidcards.yaml

resources/default-settings/sync_rfidcards.default.yaml

@@ -0,0 +1,9 @@
+sync_rfidcards:
+  on_rfid_scan_enabled: true # bool
+  mode: mount # 'mount' or 'ssh'
+  credentials:
+    server: ''
+    port:  # int
+    timeout: 1 # int
+    path: ''
+    username: ''

src/jukebox/components/gpio/gpioz/plugin/connectivity.py

@@ -19,6 +19,7 @@
 import components.gpio.gpioz.plugin
 from components.gpio.gpioz.core.output_devices import LED, PWMLED, Buzzer, TonalBuzzer, RGBLED
 from components.gpio.gpioz.core.converter import VolumeToRGB
+from components.rfid.reader import RfidCardDetectState
 
 logger = logging.getLogger('gpioz')
 
@@ -61,10 +62,10 @@ def register_rfid_callback(device):
         - :class:`components.gpio.gpioz.core.output_devices.TonalBuzzer`
     """
 
-    def rfid_callback(card_id: str, state: int):
-        if state == 0:
+    def rfid_callback(card_id: str, state: RfidCardDetectState):
+        if state == RfidCardDetectState.isRegistered:
             device.flash(on_time=0.1, n=1, tone=BUZZ_TONE)
-        elif state == 1:
+        elif state == RfidCardDetectState.isUnkown:
             device.flash(on_time=0.1, off_time=0.1, n=3, tone=BUZZ_TONE)
 
     components.rfid.reader.rfid_card_detect_callbacks.register(

src/jukebox/components/playermpd/__init__.py

@@ -95,7 +95,7 @@
 import misc
 
 from jukebox.NvManager import nv_manager
-
+from .playcontentcallback import PlayContentCallbacks, PlayCardState
 
 logger = logging.getLogger('jb.PlayerMPD')
 cfg = jukebox.cfghandler.get_handler('jukebox')
@@ -284,6 +284,12 @@ def update(self):
             state = self.mpd_client.update()
         return state
 
+    @plugs.tag
+    def update_wait(self):
+        state = self.update()
+        self._db_wait_for_update(state)
+        return state
+
     @plugs.tag
     def play(self):
         with self.mpd_lock:
@@ -444,9 +450,17 @@ def play_card(self, folder: str, recursive: bool = False):
             is_second_swipe = self.music_player_status['player_status']['last_played_folder'] == folder
         if self.second_swipe_action is not None and is_second_swipe:
             logger.debug('Calling second swipe action')
+
+            # run callbacks before second_swipe_action is invoked
+            play_card_callbacks.run_callbacks(folder, PlayCardState.secondSwipe)
+
             self.second_swipe_action()
         else:
             logger.debug('Calling first swipe action')
+
+            # run callbacks before play_folder is invoked
+            play_card_callbacks.run_callbacks(folder, PlayCardState.firstSwipe)
+
             self.play_folder(folder, recursive)
 
     @plugs.tag
@@ -585,12 +599,33 @@ def set_volume(self, volume):
             self.mpd_client.setvol(volume)
         return self.get_volume()
 
+    def _db_wait_for_update(self, update_id: int):
+        logger.debug("Waiting for update to finish")
+        while self._db_is_updating(update_id):
+            # a little throttling
+            time.sleep(0.1)
+
+    def _db_is_updating(self, update_id: int):
+        with self.mpd_lock:
+            _status = self.mpd_client.status()
+            _cur_update_id = _status.get('updating_db')
+            if _cur_update_id is not None and int(_cur_update_id) <= int(update_id):
+                return True
+            else:
+                return False
+
 
 # ---------------------------------------------------------------------------
 # Plugin Initializer / Finalizer
 # ---------------------------------------------------------------------------
 
 player_ctrl: PlayerMPD
+#: Callback handler instance for play_card events.
+#: - is executed when play_card function is called
+#: States:
+#: - See :class:`PlayCardState`
+#: See :class:`PlayContentCallbacks`
+play_card_callbacks: PlayContentCallbacks[PlayCardState]
 
 
 @plugs.initialize
@@ -599,6 +634,9 @@ def initialize():
     player_ctrl = PlayerMPD()
     plugs.register(player_ctrl, name='ctrl')
 
+    global play_card_callbacks
+    play_card_callbacks = PlayContentCallbacks[PlayCardState]('play_card_callbacks', logger, context=player_ctrl.mpd_lock)
+
     # Update mpc library
     library_update = cfg.setndefault('playermpd', 'library', 'update_on_startup', value=True)
     if library_update:

src/jukebox/components/playermpd/playcontentcallback.py

@@ -0,0 +1,37 @@
+
+from enum import Enum
+from typing import Callable, Generic, TypeVar
+
+from jukebox.callingback import CallbackHandler
+
+
+class PlayCardState(Enum):
+    firstSwipe = 0,
+    secondSwipe = 1
+
+
+STATE = TypeVar('STATE', bound=Enum)
+
+
+class PlayContentCallbacks(Generic[STATE], CallbackHandler):
+    """
+    Callbacks are executed in various play functions
+    """
+
+    def register(self, func: Callable[[str, STATE], None]):
+        """
+        Add a new callback function :attr:`func`.
+
+        Callback signature is
+
+        .. py:function:: func(folder: str, state: STATE)
+            :noindex:
+
+            :param folder: relativ path to folder to play
+            :param state: indicator of the state inside the calling
+        """
+        super().register(func)
+
+    def run_callbacks(self, folder: str, state: STATE):
+        """:meta private:"""
+        super().run_callbacks(folder, state)

src/jukebox/components/rfid/reader/__init__.py

@@ -3,6 +3,7 @@
 import time
 import importlib
 from typing import Callable
+from enum import Enum
 
 import jukebox.plugs as plugs
 import jukebox.cfghandler
@@ -20,15 +21,18 @@
 cfg_cards = jukebox.cfghandler.get_handler('cards')
 
 
-class ServiceIsRunningCallbacks(CallbackHandler):
-    """
-    Callbacks are executed when
+class RfidCardDetectState(Enum):
+    received = 0,
+    isRegistered = 1
+    isUnkown = 2
+
 
-        * valid rfid card detect
-        * unknown card detect
+class RfidCardDetectCallbacks(CallbackHandler):
+    """
+    Callbacks are executed if rfid card is detected
     """
 
-    def register(self, func: Callable[[str, int], None]):
+    def register(self, func: Callable[[str, RfidCardDetectState], None]):
         """
         Add a new callback function :attr:`func`.
 
@@ -38,18 +42,18 @@ def register(self, func: Callable[[str, int], None]):
             :noindex:
 
             :param card_id: Card ID
-            :param state: 0 if card id is registered, 1 if card id is unknown
+            :param state: See :class:`RfidCardDetectState`
         """
         super().register(func)
 
-    def run_callbacks(self, card_id: str, state: int):
+    def run_callbacks(self, card_id: str, state: RfidCardDetectState):
         """:meta private:"""
         super().run_callbacks(card_id, state)
 
 
 #: Callback handler instance for rfid_card_detect_callbacks events.
-#: See :class:`ServiceIsRunningCallbacks`
-rfid_card_detect_callbacks: ServiceIsRunningCallbacks = ServiceIsRunningCallbacks('rfid_card_detect_callbacks', log)
+#: See :class:`RfidCardDetectCallbacks`
+rfid_card_detect_callbacks: RfidCardDetectCallbacks = RfidCardDetectCallbacks('rfid_card_detect_callbacks', log)
 
 
 class CardRemovalTimerClass(threading.Thread):
@@ -175,6 +179,10 @@ def run(self):  # noqa: C901
 
                         # (3) Check if this card is in the card database
                         # TODO: This card config read is not thread safe
+
+                        # run callbacks on successfull read before card_entry is processed
+                        rfid_card_detect_callbacks.run_callbacks(card_id, RfidCardDetectState.received)
+
                         card_entry = cfg_cards.get(card_id, default=None)
                         if card_entry is not None:
 
@@ -207,12 +215,12 @@ def run(self):  # noqa: C901
                                 # dodgy cards database entry
                                 # TODO: This call happens from the reader thread, which is not necessarily what we want ...
                                 # TODO: Change to RPC call to transfer execution into main thread
-                                rfid_card_detect_callbacks.run_callbacks(card_id, 0)
+                                rfid_card_detect_callbacks.run_callbacks(card_id, RfidCardDetectState.isRegistered)
                                 plugs.call_ignore_errors(card_action['package'], card_action['plugin'], card_action['method'],
                                                          args=card_action['args'], kwargs=card_action['kwargs'])
 
                         else:
-                            rfid_card_detect_callbacks.run_callbacks(card_id, 1)
+                            rfid_card_detect_callbacks.run_callbacks(card_id, RfidCardDetectState.isUnkown)
                             self._logger.info(f"Unknown card: '{card_id}'")
                             self.publisher.send(self.topic, card_id)
                     elif self._cfg_log_ignored_cards is True:

src/jukebox/components/rpc_command_alias.py

@@ -113,6 +113,19 @@
         'method': 'start',
         'title': 'Start the stop music timer',
         'ignore_card_removal_action': True},
+    # SYNCHRONISATION
+    'sync_rfidcards_all': {
+        'package': 'sync_rfidcards',
+        'plugin': 'ctrl',
+        'method': 'sync_all',
+        'title': 'Sync all audiofiles and card entries',
+        'ignore_card_removal_action': True},
+    'sync_rfidcards_change_on_rfid_scan': {
+        'package': 'sync_rfidcards',
+        'plugin': 'ctrl',
+        'method': 'sync_change_on_rfid_scan',
+        'title': "Change activation of 'on RFID scan'",
+        'ignore_card_removal_action': True},
 }
 
 # TODO: Transfer RFID command from v2.3...