correct some eggregious errors circa chatgpt, and start the multi-sink

2023-12_usb-real-data-rates.md

@@ -4,25 +4,19 @@ I've been trying to sort out how to get enough data out of the NIST printer to g
 
 To start, I spun up some max-rate-ingestion tests, on ubuntu and windows,
 
-### In Ubuntu (Lattitude 5490): 0.4 -> 0.5 Mbits/s 
+![32](images/2023-12-20_ingest-histogram-single-source-pck-32.png)
+![128](images/2023-12-20_ingest-histogram-single-source-pck-128.png)
+![128](images/2023-12-20_ingest-histogram-single-source-pck-250.png)
 
-![32byte](images/2023-12-12_ingest-histogram-single-source-pck-32-ubuntu.png) 
-![64byte](images/2023-12-12_ingest-histogram-single-source-pck-64-ubuntu.png)
-![128byte](images/2023-12-12_ingest-histogram-single-source-pck-128-ubuntu.png)
-![250byte](images/2023-12-12_ingest-histogram-single-source-pck-250-ubuntu.png)
+So, we learn that there's no real change to the underlying data rate with increasing packet sizes, and we also learn that windows and ubuntu are not actually so different:
 
-### In Windows (XPS 15 2019): 0.6 -> 0.9 Mbits/s 
+**Warning** the data in the table below are faulty (bad Mbits/s plotting and overclocked RP2040 reporting speedier deltas than were real), but they serve to show the relative performance:
 
-![32byte](images/2023-12-12_ingest-histogram-single-source-pck-32.png)
-![64byte](images/2023-12-12_ingest-histogram-single-source-pck-64.png)
-![128byte](images/2023-12-12_ingest-histogram-single-source-pck-128.png)
-![250byte](images/2023-12-12_ingest-histogram-single-source-pck-250.png)
+| <h3>Windows</h3> | <h3>Ubuntu</h3> |
+| --- | --- |
+| ![128](images/2023-12-12_ingest-histogram-single-source-pck-128.png) | ![128](images/2023-12-12_ingest-histogram-single-source-pck-128-ubuntu.png) |
 
-### Windows Subsystem for Linux (WSL) 
-
-Another option - though I'm not sure how it will handle drivers etc - is to run this mf'er out of WSL - which I will try because I am having to install ubuntu anyways on a different machine to test on the real deal. 
-
-That requires some [additional middleware](https://github.com/dorssel/usbipd-win) that looks stateful / PITA-ish, so I will just try on a separate machine; this is meant to be simple.
+Ubuntu packets are arriving a little more spaced out than in windows - but, also, the ubuntu machine (a Lattitude 5490) was under-spec'd relatively to the windows machine (XPS 15 2019). 
 
 ### Surprisingly... 
 
@@ -30,15 +24,19 @@ Windows out performs ubuntu here. Now, there is a big difference in machines as
 
 We do also win some total bandwidth when we use bigger packet sizes, sort of as expected. 
 
-But we have a clear cap near 1Mbit/sec in terms of real performance, which FWIW is 1/12th of the USB 2.0 Full-Speed-Device reported max, so there's clearly a lot missing here. Our data-printer spec asks for about 4Mbit/sec. 
+But we have a clear cap near 0.5 Mbit/sec in terms of real performance, which FWIW is 1/24th of the USB 2.0 Full-Speed-Device reported max, so there's clearly a lot missing here. Our data-printer spec asks for about 4Mbit/sec, which is also worrying: we would need to have up to five of these devices to make that possible, and have them each working in parallel. 
+
+### Other Externalities 
 
 This is also not accounting for i.e. multiple devices, flow control, flow going down (as well as up), etc. For that I will have to improve the system / etc. 
 
+This also doesn't properly inspect whether / not there is significant performance dings due to i.e. cobs, which is [some looping python, anyways](https://github.com/cmcqueen/cobs-python/blob/main/src/cobs/cobs/_cobs_py.py) - so, maybe there is real evidence that we want to i.e. ethernet to the first thing, etc. 
+
 ## 2023 12 20 
 
 So, we want to... consume and produce data, as fast as possible in either direction, and open multiple ports.
 
-I think I will get some devices w/ OLEDs on 'em, and then also do the due dilly of checking if I2C display writes are blocking or not? 
+I think that the multiple ports thing is going to teach me what I want to know about asyncio, and is simple enough that I can try to replicate it with multiprocessing as well. 
 
 
 --- 

code/serial_multi_sink/cobs_usb_serial.py

+from cobs import cobs
+import serial
+
+
+class CobsUsbSerial:
+    def __init__(self, port, baudrate=115200):
+        self.port = port
+        self.ser = serial.Serial(port, baudrate=baudrate, timeout=1)
+
+    def write(self, data: bytes):
+        data_enc = cobs.encode(data) + b"\x00"
+        self.ser.write(data_enc)
+
+    def read(self):
+        data_enc = self.ser.read_until(b"\x00")
+        data = cobs.decode(data_enc[:-1])
+        return data

code/serial_multi_sink/multi_sink.py

+from cobs_usb_serial import CobsUsbSerial 
+import asyncio 
+import struct 
+import numpy as np 
+import pandas as pd 
+import matplotlib.pyplot as plt 
+
+pck_len = 32
+
+async def read_ser(cobs_serial, rx_func):
+  while True:
+    data = cobs_serial.read()
+    if len(data) == pck_len:
+      stamp = struct.unpack('=I', data[:4])
+      rx_func(stamp)
+    await asyncio.sleep(0)
+
+def port_handler(stamp):
+  print(stamp)
+
+async def main():
+  port_one = CobsUsbSerial("COM23") 
+
+  task_one = asyncio.create_task(read_ser(port_one, port_handler))
+
+  await asyncio.gather(task_one)
+
+asyncio.run(main())
+
+# from cobs_usb_serial import CobsUsbSerial 
+# import struct 
+# import numpy as np 
+# import pandas as pd 
+# import matplotlib.pyplot as plt 
+
+# ser = CobsUsbSerial("COM23") 
+
+# stamp_count = 1000
+# pck_len = 250
+
+# stamps = np.zeros(stamp_count)
+
+# for i in range(stamp_count):
+#   bts = ser.read()
+#   if len(bts) == pck_len:
+#     stamp = struct.unpack('=I', bts[:4])
+#     stamps[i] = stamp[0]
+
+# print("stamps, ", stamps)
+
+# df = pd.DataFrame({'timestamps': stamps})
+
+# df['deltas'] = df['timestamps'].diff() 
+
+# # clean NaN's 
+# df = df.dropna()
+
+# # wipe obviously-wrong deltas (i.e. the 1st, which goes 0-start-us) 
+# df = df[df['deltas'] < 100000]
+
+# # Plotting
+# fig, ax1 = plt.subplots(figsize=(11, 5))
+
+# # Primary x-axis (time deltas)
+# df['deltas'].plot(kind='hist', bins=100, ax=ax1)
+# ax1.set_xlabel('Time-Stamp Deltas (us)')
+# ax1.set_ylabel(f'Frequency (of {stamp_count})')
+
+# # Secondary x-axis (bandwidth)
+# ax2 = ax1.twiny()
+# ax2.set_xlabel('Estimated Bandwidth (Mbits/s)')
+
+# # Set the limits of the secondary axis based on the primary axis
+# # new_tick_locations = np.linspace(df['deltas'].min(), df['deltas'].max(), num=len(ax1.get_xticks()))
+
+# # Convert tick locations to bandwidth
+# # bandwidths = [(pck_len * 8) * (1e6 / x) for x in new_tick_locations]
+
+# x_ticks = ax1.get_xticks()
+# bandwidth_ticks = [((pck_len * 8) * (1e6 / x)) / 1e6 for x in x_ticks]
+
+# ax2.set_xlim(max(bandwidth_ticks), min(bandwidth_ticks))
+
+# plt.title(f'Single-Source COBS Data Sink Deltas, pck_len={pck_len}')
+
+# plt.tight_layout()
+
+# plt.show()

code/serial_multi_sink/serial_list.py

+import serial.tools.list_ports
+
+def list_serial_ports():
+    ports = serial.tools.list_ports.comports()
+    for port in ports:
+        print(f"Port: {port.device}")
+        print(f" - Description: {port.description}")
+        if port.serial_number:
+            print(f" - Serial Number: {port.serial_number}")
+        if port.manufacturer:
+            print(f" - Manufacturer: {port.manufacturer}")
+        if port.product:
+            print(f" - Product: {port.product}")
+        if port.vid is not None:
+            print(f" - VID: {port.vid:04X}")
+        if port.pid is not None:
+            print(f" - PID: {port.pid:04X}")
+        print()
+
+list_serial_ports()

code/serial_sink/sink.py

@@ -4,10 +4,10 @@ import numpy as np
 import pandas as pd 
 import matplotlib.pyplot as plt 
 
-ser = CobsUsbSerial("/dev/ttyACM0") 
+ser = CobsUsbSerial("COM23") 
 
-stamp_count = 10000
-pck_len = 32
+stamp_count = 1000
+pck_len = 250
 
 stamps = np.zeros(stamp_count)
 
@@ -29,10 +29,6 @@ df = df.dropna()
 # wipe obviously-wrong deltas (i.e. the 1st, which goes 0-start-us) 
 df = df[df['deltas'] < 100000]
 
-# Bandwidth calculation (bytes/us to Mbits/s)
-# 1 byte/us = 8 Mbits/s
-df['bandwidth'] = (pck_len / df['deltas']) * 8 * 1e3  # Convert to Mbits/s
-
 # Plotting
 fig, ax1 = plt.subplots(figsize=(11, 5))
 
@@ -46,10 +42,15 @@ ax2 = ax1.twiny()
 ax2.set_xlabel('Estimated Bandwidth (Mbits/s)')
 
 # Set the limits of the secondary axis based on the primary axis
-new_tick_locations = np.linspace(df['deltas'].min(), df['deltas'].max(), num=len(ax1.get_xticks()))
+# new_tick_locations = np.linspace(df['deltas'].min(), df['deltas'].max(), num=len(ax1.get_xticks()))
 
 # Convert tick locations to bandwidth
-# ax2.set_xlim([pck_len / x * 8 * 1e3 for x in new_tick_locations])
+# bandwidths = [(pck_len * 8) * (1e6 / x) for x in new_tick_locations]
+
+x_ticks = ax1.get_xticks()
+bandwidth_ticks = [((pck_len * 8) * (1e6 / x)) / 1e6 for x in x_ticks]
+
+ax2.set_xlim(max(bandwidth_ticks), min(bandwidth_ticks))
 
 plt.title(f'Single-Source COBS Data Sink Deltas, pck_len={pck_len}')
 

code/serial_source/serial_source.ino

@@ -40,7 +40,7 @@ void loop() {
   // tx a stamp AFAP 
   if(cobs.clearToSend()){
     chunk.u = micros();
-    cobs.send(chunk.bytes, 32);
+    cobs.send(chunk.bytes, 250);
     digitalWrite(PIN_LED_G, !digitalRead(PIN_LED_G));
   }
   // blink to see hangups