unicorn-binance-depth-cache-cluster Change Log

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

Discussions about unicorn-binance-depth-cache-cluster releases!

How to upgrade to the latest version!

0.9.1.dev (development stage/unreleased/unstable)

Added

• Helm chart: optional N DCN pods per node — the optimal distribution is one DCN per CPU core. The default is unchanged (a DaemonSet, exactly 1 DCN pod per node). Setting dcn.coresPerNode > 1 switches the DCN to a Deployment that auto-detects the count of schedulable, non-control-plane nodes via a cluster lookup and renders replicas = coresPerNode × nodeCount, spread evenly across nodes with topologySpreadConstraints (maxSkew: 1).

• Helm values dcn.coresPerNode, dcn.nodeCount (override auto-detection), dcn.replicas (force Deployment mode + override the whole calculation), dcn.nodeSelector (pin DCN to a node pool) and dcn.resources (set requests == limits for Guaranteed QoS / per-core CPU pinning).

• Helm values restapi.perNode, restapi.nodeCount, restapi.replicas, restapi.nodeSelector and restapi.resources — symmetric to the dcn.* block, for tuning the RESTAPI topology.

Changed

• Helm chart: RESTAPI is now a DaemonSet by default (exactly 1 RESTAPI pod per node, auto-scales with the cluster) instead of a hard-coded StatefulSet with replicas: 3 + per-host podAntiAffinity. The old layout broke on clusters with < 3 nodes (pods stuck Pending) and wasted capacity on clusters with > 3 nodes. Setting restapi.perNode > 1 (or an explicit restapi.replicas) switches RESTAPI to a Deployment with replicas = perNode × nodeCount, spread evenly via topologySpreadConstraints. The replicaCount.restapi value is removed.

• admin/k8s/ubdcc-restapi.yaml updated to match (plain DaemonSet).

0.9.1

Fixed

• get_k8s_runtime_information() in packages/ubdcc-shared-modules/ubdcc_shared_modules/App.py: the in-cluster Kubernetes API client lost its Bearer token when UBDCC_K8S_VERIFY_SSL=false was set, causing every API request to hit the API server as system:anonymous and fall back to dev mode (random pod identity + localhost:42080 mgmt routing). The previous flow loaded the in-cluster config, then created a copy via Configuration.get_default_copy(), mutated verify_ssl=False on the copy, and pushed it back via Configuration.set_default(); in the pinned kubernetes-client version this swap dropped the api_key['authorization'] Bearer header. New flow: pre-create a kubernetes.client.Configuration(), set verify_ssl=False on it before calling kubernetes.config.load_incluster_config(client_configuration=cfg) so the lib writes the token directly into the already-modified Configuration, then instantiate CoreV1Api / CustomObjectsApi with an explicit ApiClient(configuration=cfg) — no more global default-swap. Verified out-of-band with a raw urllib request from inside the pod (HTTP 200, real pod JSON) on a Vultr Kubernetes cluster where the previous code path returned 403.

0.9.0

Added

• UBDCC_K8S_VERIFY_SSL environment variable (default "true") for the three services (ubdcc-mgmt, ubdcc-restapi, ubdcc-dcn). Set to "false" to disable TLS verification for the in-cluster Kubernetes API calls made by get_k8s_runtime_information() / get_k8s_nodes() in packages/ubdcc-shared-modules/ubdcc_shared_modules/App.py. Intended as an opt-out for clusters that ship Kubernetes API server certificates rejected by strict X.509 verification (e.g. missing AuthorityKeyIdentifier, keyCertSign set on a non-CA cert — observed on some managed clusters). The flag is wired through the Helm chart (dev/helm/ubdcc/values.yaml: k8s.verifySsl) and the raw Kubernetes manifests in admin/k8s/. When set to false, a WARNING is logged on every service start and urllib3’s InsecureRequestWarning is silenced.

0.8.1

Hotfix release — only the ubdcc CLI meta-package was bumped; the service packages (ubdcc-mgmt, ubdcc-restapi, ubdcc-dcn, ubdcc-shared-modules) stay at 0.8.0 because they are unchanged. The CLI pins them with ==0.8.0.

Fixed

• ubdcc start on Windows: packages/ubdcc/ubdcc/cli.py now embeds cwd and log_level via repr() ({cwd!r}) into the python -c command used to spawn mgmt / restapi / dcn services. The previous '{cwd}' f-string interpolation emitted raw backslashes into the child Python source, so a working directory like c:\ubdcc-test was parsed with \u as a Unicode escape (\ubdcc -> U+BDCC), pointing the child at a non-existent path and crashing os.chdir() with FileNotFoundError [WinError 2]. Linux/macOS were unaffected because POSIX paths have no backslashes.

0.8.0

Changed

• Default service log level lowered from DEBUG to ERROR in packages/ubdcc-shared-modules/ubdcc_shared_modules/App.py. Cuts disk usage in production drastically — ~/.unicorn-binance-suite/logs/ubdcc-*.log no longer fills up with per-tick DEBUG noise. Affects every service (mgmt, restapi, dcn) regardless of how it’s launched (ubdcc start, K8s, manual).

Added

• packages/ubdcc: ships ubdcc-dashboard >= 0.2.0 as a runtime dependency (setup.py, requirements.txt, pyproject.toml). pip install ubdcc now also installs the browser-based UBDCC Dashboard — launch it with ubdcc-dashboard start. README updated accordingly.

• ubdcc start --log-level DEBUG|INFO|WARNING|ERROR|CRITICAL to pick a log level for the spawned mgmt / restapi / dcn services without having to edit code. Propagated as a log_level constructor kwarg through Mgmt, RestApi, DepthCacheNode → ServiceBase → App.

0.7.0

Changed

• Breaking: dropped the redundant ubdcc_ prefix from the three user-facing credential endpoints to align with the rest of the REST API (/create_depthcache, /get_asks, …):

  • /ubdcc_add_credentials → /add_credentials

  • /ubdcc_remove_credentials → /remove_credentials

  • /ubdcc_get_credentials_list → /get_credentials_list Renamed on both the public restapi and the internal mgmt service. Internal cluster endpoints (/ubdcc_assign_credentials, /ubdcc_node_*, /ubdcc_update_depthcache_distribution, …) keep their prefix — it signals “not a public API”. The matching UBLDC cluster client methods were renamed too (requires UBLDC ≥ 2.14.0): use cluster.add_credentials() / cluster.remove_credentials() / cluster.get_credentials_list(). The ubdcc credentials add/list/remove CLI continues to work unchanged (it now hits the new endpoints internally).

• packages/ubdcc-dcn: bumped unicorn-binance-local-depth-cache minimum to >=2.14.0 across setup.py, requirements.txt and pyproject.toml — the renamed cluster client methods live there.

Fixed

• AccountGroups.py: binance.com-margin-testnet and binance.com-isolated_margin-testnet are now mapped to the binance.com-testnet account group. Without this, credential assignment (get_account_group()) returned None for testnet-margin DCNs, so no API keys could be dispatched to them. Spot-testnet and margin-testnet share the same testnet.binance.vision credentials, so they belong in the same account group.

0.6.0

Added

• CLI status: new “DC restarts” section lists every depth cache whose WebSocket stream has been restarted at least once, sorted by restart count descending, with a human-readable “last restart” timestamp (e.g. 45s ago, 2h ago). Hidden when no DC has restarted.

• CLI --help: the top-level help now lists the credentials subcommands (add, remove, list) in the epilog alongside the interactive shell commands, so users no longer need to run ubdcc credentials --help to discover them.

Changed

• Credential assignment is now event-driven and self-healing. Previously, a DCN pulled its API key exactly once (on the first depth cache it created for a given exchange) and cached the result — including a sticky None after any transient failure, leaving assigned_dcns: [] forever across cluster restarts. Now:

  • Database.rebalance_account_group() redistributes every active DCN round-robin across the credentials available for that account group. It runs on add_credentials() / delete_credentials() and as part of revise() when the DCN population changes.

  • DCNs cache only the credential_id per account group as a comparison reference. On every main-loop tick they ask mgmt for their current assignment and, when the id differs, hot-swap the UBRA in every affected BinanceLocalDepthCacheManager via the new set_credentials() public API (requires UBLDC ≥ 2.13.0). WebSocket streams keep running; new credentials take effect from the next REST call (snapshot / resync).

• packages/ubdcc-dcn: bumped unicorn-binance-local-depth-cache minimum to >=2.13.0 across setup.py, requirements.txt and pyproject.toml for the set_credentials() method.

Fixed

• mgmt /ubdcc_update_depthcache_distribution: last_restart_time was parsed from the query string but never forwarded to Database.update_depthcache_distribution() — so the RESTARTS counter per DC distribution entry stayed at 0 forever, no matter how often a UBLDC stream restarted. Also: the query param arrived as a string but the DB expected a float, so the subsequent != compare against the stored value would have been string-vs-float. Both are fixed now: cast to float with an explicit error response (#1024) on malformed input, and pass the value through to the DB call.

0.5.0

Added

• CLI status: show UBLDC version in parentheses next to each DCN’s version

• CLI status: DepthCache redundancy summary — replica breakdown (running/starting) and redundancy categories (fully redundant, degraded, no redundancy)

• AccountGroups: binance.com-vanilla-options → binance.com, binance.com-vanilla-options-testnet → binance.com-futures-testnet

0.4.1

Fixed

• Compiled file ‘ubdcc_shared_modules.AccountGroups’ was missing in the wheels

• Pinning ubdcc versions in ubdcc cli requirements

0.4.0

Added

• Binance API credentials support. Optional: add one or more (api_key, api_secret) pairs per account group (binance.com, binance.com-testnet, binance.com-futures-testnet, binance.us, binance.tr) via new REST endpoints /ubdcc_add_credentials (POST/GET), /ubdcc_remove_credentials, /ubdcc_get_credentials_list and matching ubdcc credentials add/list/remove CLI subcommands. Mgmt load-balances credentials across DCN pods (fewest-assigned-first); DCNs fetch their assigned key via the internal /ubdcc_assign_credentials endpoint and pass it to UBLDC as a BinanceRestApiManager. Lets the cluster use authenticated rate limits without mandating credentials — public-only operation remains unchanged. Keys are part of the self-healing DB sync (full cleartext cluster-internal); public responses only show masked previews and the list of assigned DCN UIDs per key. See README → API Credentials for the security trade-offs.

• get_asks / get_bids: when failover recovers from one or more unreachable DCNs, the successful response now includes an error_id (#5001) and an error message listing the pods that failed before success. Gives monitoring a visible signal that failover occurred without changing the OK-with-data semantics. Closes #3.

• DepthCache distribution tracking: new DB fields LAST_DISTRIBUTION_CHANGE + DISTRIBUTION_CHANGES (top-level, counts how often a DC has been re-assigned across pods) and RESTARTS per distribution entry (counts stream restarts on the currently assigned pod). Visible in get_depthcache_info / get_depthcache_list — useful for diagnosing upstream stability and cluster rebalancing history. Closes #1.

Removed

• External ubdcc restart <name> CLI subcommand — it didn’’’t actually work because the spawn functions live in the interactive shell closure. Restart is still available inside the interactive ubdcc> shell.

Fixed

• DCN startup crash: self._restart_queue was initialized after super().__init__(), but ServiceBase.__init__() blocks on self.app.start() and runs main() synchronously — so the assignment was never reached. Every DCN crashed seconds after registering with AttributeError: '_restart_queue', leaving only mgmt and restapi alive. Queue init now runs before super().__init__().

0.3.3

Added

• New ubdcc meta-package and cluster manager (pip install ubdcc)

• ubdcc CLI command with interactive console:

  • ubdcc start --dcn 4 — starts mgmt, restapi and DCN processes

  • ubdcc status — shows all pods with role, name, port, status

  • ubdcc stop — graceful shutdown via REST

  • ubdcc restart <name> — full restart (shutdown + respawn) for mgmt, restapi or DCN

  • add-dcn [count] — spawn new DCN process(es) from the interactive shell

  • remove-dcn <count|name> — stop DCN(s) by count or by name

  • Interactive ubdcc> prompt during start for live management

  • python -m ubdcc entry point for running from source during development

• /shutdown REST endpoint on all pods (dev-mode only) for graceful process shutdown

• /create_depthcaches now supports both GET (comma-separated markets) and POST (JSON body)

• mgmt_port parameter on all services — allows custom port configuration via CLI

• .ubdcc state file for automatic port detection across CLI commands

• debug=true now includes post_body in response for POST requests

• Port retry on bind failure — fixes race condition when multiple DCNs start simultaneously

• Mermaid architecture diagram in README (replaces external Lucidchart image)

• Comprehensive README rewrite: What is UBDCC, Why, Architecture, Local Setup, REST API reference with all endpoints, debugging guide

• Python 3.9-3.14 support on Linux, macOS and Windows (all 5 packages)

• Build workflows updated to UBWA pattern: 3 OS matrix, cibuildwheel v3.4.1, sdist, GitHub Release

• New build workflow for ubdcc package (pure Python)

Changed

• Replaced json with orjson for faster JSON parsing (closes #5)

• /create_depthcaches switched from GET to POST as primary method (closes #10)

• Build target: Python 3.14 only → Python 3.9-3.14 on all platforms

• Aligned project_urls across all packages with UBS standard

Fixed

• Remove remaining LUCIT references in issue templates, PR template, container LICENSE files, container READMEs, dev scripts

• Fix issue templates: correct project name, add Python 3.13/3.14 to dropdown

• Fix README: wrong port (42080 → 42081), old project name, typos

• Fix double JSON serialization in UBLDC cluster POST requests

• /shutdown endpoint now force-exits via os._exit(0) 0.5s after the response — previously processes could hang if the main loop was stuck in asyncio.sleep

• is_port_free() now sets SO_REUSEADDR — previously the TIME_WAIT state after a restart caused the new mgmt to land on the next port, breaking restapi/DCN connections

Removed

• License purchase and commercial support sections from all package READMEs

• External Lucidchart dependency (replaced with inline Mermaid diagram)

0.2.0

Added

• AGENTS.md and TASKS.md

Changed

• Rebranded from LUCIT to open source MIT

• Renamed packages: lucit-ubdcc-* → ubdcc-*, namespaces: lucit_ubdcc_* → ubdcc_*

• Kubernetes namespace: lucit-ubdcc → ubdcc

• LICENSE: LSOSL → MIT

• Author: LUCIT Systems and Development → Oliver Zehentleitner

• All URLs updated from lucit.tech/shop.lucit.services to GitHub

• Sphinx docs theme: python_docs_theme_lucit → alabaster

Fixed

• create_depth_cache() → create_depthcache() in DepthCacheNode.py (deprecated method call)

• Per-package LICENSE files replaced with correct MIT license

• Remaining LUCIT references in file headers, pyproject.toml descriptions, and per-package CHANGELOGs

• UBLDC dependency pin: unicorn_binance_local_depth_cache==2.6.0 → unicorn-binance-local-depth-cache>=2.8.1

Removed

• LUCIT licensing system (LicensingManager, LicensingExceptions, submit_license endpoints)

• lucit-licensing-python dependency from all packages

• License params (lucit_api_secret, lucit_license_token) from UBLDC instantiation in DCN

0.1.4

Fixed

• https://github.com/oliver-zehentleitner/unicorn-binance-depth-cache-cluster/issues/8 - We don’t overwrite the None anymore.

0.1.3

MVP release!