The Versatile Interface Adapter
Even though the MC3 is in its core a Motorola 68xx design it can use 65xx parts. The bus and timings are the same. I initially planed on building an I/O board around the 6321 PIA but ended up trying out the feature packed 6522 VIA instead. The 6522 is actually still being manufactured by WDC! The 6522 is familiar to anyone working with the 6502 or MOS/Commodore machines. In general it's a souped up 6821 with two timers and a shift register. The 6522 consumes more address space than the 6821 but each register can be directly addressed, thus making it faster for certain tasks. It requires four address pins instead of only two but for me that's a perfect match since my I/O boards each have 4 address lines making it a 1:1 match. Connecting the 6522 to the MC3 20-pin I/O expansion bus require no extra circuitry. It's only a matter of connecting the corresponding pins together. Since there is no IRQ line on the I/O bus I will have to settle for polling the 6522. Thank you Douglas Beattie for sharing your Eagle 65xx library. The 40-pin chip is a tight fit on the small 20-pin I/O board but it worked. For now there are no connectors for the 6522 ports. Not much space left but I should be able to mount a suitable connector at the top of the board. I really like how the 6522 engineers have separated the bus pins and the I/O pins to different sides of the chip.

Inside the 6522

Above is the block diagram as seen in the G65SC22 datasheet. The 6522 has two 8-bit ports with two handshake lines each. Both ports are controlled by their corresponding data register and data direction register (just like the 6821 and many other designs based on the same concept). It also has two 16-bit timers and a shift register. Care must be taken to avoid the infamous shift register bug happening in mode 011, external clocking of the shift register. The bug occurs when the shift register clock input falls at about the same time as the Φ2 clock falls, making the shift register lose one data bit. Very strange framing errors can occur when it happens. Garth Wilson has proposed a neat solution for this that involves an external latch. Rumors has it that some variants of the 6522 does not suffer from this bug. I have a used the G65SC22 from CMD in my design that supposedly have this bug fixed. I need to set up a test to see if this is the case. In general, when dealing with the 6522 you have to assume all of them contains the bug in mode 011. The other shift register modes are unaffected as far as I know.
A bigger case for the MC3
The MC3 had expanded way beyond the initial enclosure and really needed to move into something bigger. Having exposed boards feels risky. Especially with these scarce old components. I really do not want to accidentally kill anything. Also I did not want to just put everything in a standard boring electronic box. The all CMOS based MC3 does not generate much heat so active cooling is not needed but a well ventilated case also ables for some peeking inside which is fun. Isolation to reduce RFI and protect from ESD was also added to the requirement specification. It looked like I would need to design something from scratch. Old wooden TV-sets have always fascinated me. Dad worked as a TV repairman many years ago and as a kid I remember watching him working with those old wooden TV's. That's my inspiration for the MC3 enclosure. Time to warm up the Festools and get cutting! First off I decided to use edge glued oak as the base for the enclosure. It makes it all pretty robust and also look a bit more furniture-ish. The construction is a plain wooden frame with perforated side panels of aluminum. Those panels are actually ventilated sides made for Schroff 19" enclosures that I found in the scrap pile at work. Oak does not provide any ESD protection or RFI isolation to speak of so I dressed the inside of the wooden parts with aluminum tape. The front side of the box has no buttons for a clean look. Since I use the RS-232 reset circuit I very rarely need to access the physical reset switch anyway. On the back there is a cut-out in the wood for an access panel containing all connectors and buttons. A view of the back panel. Some space left for expansion. Both aluminum side panels can be removed. The main boards are stacked and screwed to the bottom with screws that go all the way through. Even when the aluminum sides are on it's possible to get a glimpse of what's inside. Also visible here is the aluminum tape on the wooden parts. Boards mounted here are from top to bottom; I/O expansion board with a few mini I/O boards on top, video board, CPU board and extended RAM board. The two boards on the bottom are empty for further expansion. Now it feels safe to let the cat stroll around the computer again!
Upgrading Debian Wheezy to Jessie
Jessie has reached stable and for those of you who don't want to reinstall your system the upgrade procedure is quite straight forward. As always make sure you have a backup before you start in case something goes wrong. First step is to make sure you have no special packages installed. Packages from the standard Debian repositories should be safe as well as most of the backports but I strongly recommend to remove packages from third party sources before starting. You also have to be logged in as root to perform the update. Step 1 - Make sure your Wheezy system is up to date This is done using the regular apt upgrade procedures. # apt-get update # apt-get upgrade # apt-get dist-upgrade Also make sure you have the latest keyrings installed. # apt-get install debian-keyring debian-archive-keyring Step 2 - Update apt sources and change wheezy to jessie Edit your /etc/apt/sources.list and change all 'wheezy' to 'jessie'. Also don't forget to take a look at the files in /etc/apt/sources.list.d/ (if there are any) and perform the same replace procedure. Step 3 - Disable X Since a lot of packages will be updated and daemons will be restarted there is a potential risk of screwing something up when running under X. Go to a regular TTY console and continue the upgrade from there. Alt+F1 or Alt+Ctrl+F1 should get you to the first console. Log in as root and continue to the next step. Step 4 - Perform the actual upgrade When you are logged in to the TTY console. The upgrade process may be initiated. First we must update the latest sources. # apt-get clean # apt-get update The next two steps is the actual upgrade that will alter the system. First we perform a normal upgrade. A lot of packages will be replaced and also quite a lot will be kept back. This is normal. # apt-get upgrade This is the first step of the actual upgrade. The installer may ask questions that I can not foresee here but as long as you are in a TTY console it should be safe to restart daemons etc. Also there may be questions about configuration files that is edited by the user and set to be replaced by the upgrade. If unsure, sticking to the default value is often a safe choice. If the screen goes back to X press Alt+F1 (or Alt+Ctrl+F1) to get back to the install console. When the first upgrade step is complete we can go on to the next one. # apt-get dist-upgrade This step will also upgrade those packages earlier reported as kept back. Step 5 - Reboot into the new system # reboot This is it. I have performed this upgrade on several systems now with good results as long as no third party packages or user modifications screws something up. As always, to get the cleanest possible system the only way is to start from scratch and do a clean install from the new installation media. A note about systemd Debian Jessie is shipped with systemd. Yes, that's true. Without getting into a discussion about systemd itself I will just add that it is possible to remove systemd from jessie (well, most of it at least). See this link for more info. I will post the instructions here directly from the link above in case something happens. First install the SysV init packages # apt-get install sysvinit-core sysvinit sysvinit-utils Then reboot your machine and remove all of the systemd packages. BE AWARE that the following command removes packages that depend on systemd itself or things like libpam-systemd! # apt-get remove --purge --auto-remove systemd Prevent apt from installing systemd packages in the future. # echo -e 'Package: systemd\nPin: origin ""\nPin-Priority: -1' > /etc/apt/preferences.d/systemd Prevent apt from installing any systemd like packages in the future (note the star before and after systemd) # echo -e 'Package: *systemd*\nPin: origin ""\nPin-Priority: -1' > /etc/apt/preferences.d/systemd
Panasonic DMC-G6 hot pixel removal
After growing tired of lugging my big and bulky DSLR camera around everywhere I began looking into smaller alternatives and picked up a Panasonic DMC-G6. A relatively cheap mirror-less MFT camera. The GH2/GH3/GH4 series cameras from Panasonic have proven to be quite successful both for photo and video so I figured the G6 might be a good and cheap starting model to get to know the Panasonic world a bit. The G6 is similar to the GH4 with regards to handling and menus etc. After using it for over six months I must say that I'm quite pleased. It's performing very well an has a lot of features I would rather expect to see in a higher-end camera. It comes at a somewhat buggy price though but the only really serious bug I've encountered so far is in the "Pixel Refresh" feature. It annoyed me for a long time but now I may finally have found a solution.

Image sensors and pixels

Every CMOS and CCD image sensor today have random faults. Even the high-end ones. All pixels are not equally sensitive and some may even be completely dead and locked at a full black or full white level. Every sensor manufacturer has a threshold on how many errors are acceptable in production. Sensors that does not reach this threshold are trashed and it is safe to assume that no sensor have 100% working pixels. Especially over time since age and radiation will create more defects. The most devestating defect is when pixels are too bright. They are easily spotted by the human eye. This is what is commonly referred to as hot pixels or stars. Camera manufacturers use software algorithms to calibrate each sensor individually to compensate for pixel defects. Some camera models require a service technician and/or special tools to perform this calibration and some models can perform the calibration in-camera, either initiated by the user or automatically.

Discovering the Pixel Refresh function

Panasonic have decided not to hide the sensor calibration feature from the user and they call it "Pixel Refresh". The details around this function is very vague and the manual is not very helpful. The manual does however say the following: "Imaging device and image processing is optimised when the camera is purchased. Use this function when bright spots that do not exist in the subject get recorded." Okay, Pixel Refresh is clearly meant to hide hot pixels. That's great! Eventually a few bright pixels showed up on my video footage and I decided it was time to try out the Pixel Refresh feature and YIKES! After this all my pictures and videos looked like the milky way! I immediately realized that something must have gone very very wrong. There were several extremely bright pixels clearly visible even at ISO 200. What was even more baffling was that the hot pixels moved around when changing the sensitivity/ISO and some pixels were only visible on the LCD/WF and some were only visible on the recorded media. Running Pixel Refresh again did not really improve the situation but after countless other attempts I really began to suspect that this had to be a software error. Spending a several hours factory resetting, updating firmware (to v1.2) and speaking to Panasonic tech support did not help either. Eventually Panasonic called for a warranty repair and after a few weeks I got my camera back with a new sensor and even more hot pixels. The service center clearly did not get the situation at all...

My solution

Almost ready to give up on the G6 I started to methodically execute the Pixel Refresh function after placing the camera in different operating modes and settings. I eventually found a combination that actually worked! The most important lesson learned here is that camera settings severely effects the Pixel Refresh function. In general, when executed, the function will display a progress bar. If this progress bar does not progress smoothly and ends with a big jump to 100% it means that the process has failed in one way or another. It took a lot of trial but here is my procedure for running Pixel Refresh on the G6 that actually provides usable results. - Let the camera reach operating temperature - Enter manual photo mode "M" (not manual video mode!!) - Turn electronic shutter ON - Set ISO to 1600 - Put lens cap on (keep stray light away from the sensor) After performing the steps above run Pixel Refresh. The process may need to be repeated in order to get rid of all hot pixels. The procedure above has worked several times now on my G6. I can not tell if it will be the same for other G6's or other Panasonic cameras but it would be interesting to know.
Reading FLEX disk images
In my quest for better knowledge of the FLEX disk operating system for the 6800 and 6809 I ended up needing to read old floppies and disk images. Images usually have the *.dsk extension and contains a raw dump of every sector on a disk. No header or other information. I found a few tools that could read raw FLEX images but they were mostly written for Windows and without source. Therefore I began digging into the FLEX disk structure with the goal of creating my own tool that could read raw FLEX images and extract individual files.

FLEX disk layout

FLEX has a hard coded sector size of 256 bytes. Addressing sectors are done by track and sector numbers. Maximum number of tracks are 256 and maximum number of sectors per track are 256, which means a single FLEX file system can be up to 256^3 bytes which is about 16MB raw capacity. Not very big by todays standards but 30 years ago this was huge. The sectors are linked together like linked lists where the first two bytes of each sector is a pointer to the next sector in the chain (actually some sectors on track 00 are exceptions from this but for the rest of the disk this is true). The first two bytes of every directory sector and file data sector contains the track and address of the next sector in the chain. End of chain is marked by setting track and sector to zero. A typical FLEX disk layout for a 40 track floppy with 20 sectors on each track: TRACK 00 SECTOR 00 --- Boot sector TRACK 00 SECTOR 01 --- Boot sector TRACK 00 SECTOR 03 --- System Information Record (SIR) TRACK 00 SECTOR 04 --- Not used TRACK 00 SECTOR 05 --- Start of directory . . TRACK 00 SECTOR 20 --- End of directory TRACK 01 SECTOR 01 --- Start of file data . . TRACK 39 SECTOR 20 --- End of file data (last sector on disk) The first sectors are boot sectors that contain executable code for loading the FLEX kernel from disk and booting the system. They can be ignored by the image reader and non-bootable disks. Tracks appears to be numbered from zero and up, and sectors from one and up. The boot sectors are exceptions from this and appears to be numbered 00 and 01. Track 00 has no sector 02. The first important sector is the System Information Record (SIR). The SIR contains basic information about the disk structure. After the SIR follows the directory sectors. The directory sectors are linked together to form one chain and the file data sectors are linked together to form another chain. The length of the directory can be varied and can span over multiple tracks if needed. The size of the directory is however determined when initializing the disk and it can usually not be changed at a later time. Below is the structure of the System Information Record (SIR). 11 byte --- Volume label 2 byte --- Volume number 1 byte --- First free track 1 byte --- First free sector 1 byte --- Last free track 1 byte --- Last free sector 2 byte --- Number of free sectors 1 byte --- Date month 1 byte --- Date day 1 byte --- Date year 1 byte --- End track 1 byte --- End sector The SIR structure is 24 bytes long and starts at byte 17 of the SIR sector. The first 16 bytes of the SIR sector is not used. The directory structure (DIR) follows a similar pattern. 8 byte --- File name 3 byte --- File extension 2 byte --- Not used 1 byte --- Start track 1 byte --- Start sector 1 byte --- End track 1 byte --- End sector 2 byte --- Total number of sectors 1 byte --- Random file flag 1 byte --- Not used 1 byte --- Date month 1 byte --- Date day 1 byte --- Date year The DIR structure is also 24 bytes long. Just as with the SIR sector the first 16 bytes of every directory sector is not used. The DIR structure is repeated 10 times for each directory sector until the end of the sector chain has been reached (all directory sectors does have the 16 byte padding at the start). One thing that had me puzzled for a while is how FLEX actually marks a file as "deleted" in the directory. After some trial and error I have discovered that deleted file entries are marked by setting the file first byte of the file name to either 0x00 or 0xFF. Another thing that had me scratching my head was the flag byte. It is used for FLEX "Random files". I have not implemented support for this file type so the flag is simply ignored but displayed. Each DIR structure contains a pointer to the first sector of the file it represents. Reading a file is a matter of simply walking the sector chain starting at the sector mentioned in the DIR structure. Each data sector has the structure below. 1 byte --- Next track 1 byte --- Next sector 2 byte --- Sector sequence number 252 byte --- File data Every sector contains a pointer to the next sector in the file, a sequence number and 252 bytes of actual file data. End of chain (and end of file) is marked by setting next track and next sector values to zero. The sequence number is a 16 bit value starting at 1 and is increased by 1 for every sector in the file. It can be used as as simple verification that the file chain appears to be correct. The last sequence number on the last sector of the file should have the same value as the total sectors value of the DIR structure.

The FLEXTRACT program

With this knowledge I began writing my FLEX image file extractor. I call it FLEXTRACT. It's a single C file that can be compiled under GCC. Source code: flextract.c Compilation and installation under Linux: $ gcc flextract.c -o flextract $ sudo cp flextract /usr/local/bin/ Running the program without options displays basic usage information: $ flextract Usage: flextract [options] [file name] [output file] Options: v - Verbose l - List directory 1 - List directory as single column x - Extract file t - Do FLEX text to ASCII conversion d - Print SIR/DIR sector dumps Output file "-" means console (stdout) Version 1.5 by Daniel Tufvesson 2015-2020 Displaying image contents: $ flextract image.dsk Image size is 204800 bytes - 40 tracks, 20 sectors/track Volume label DISK01 Volume number 0000 Free area t2 s14 - t39 s20 Free sectors 766 End sector t39 s20 Creation date 83-02-23 NAME START END SIZE DATE FLAG FILE1.BIN t01 s01 - t02 s06 26 83-02-23 00 FILE2.TXT t02 s07 - t02 s13 7 83-02-23 00 $ Extracting the binary file "FILE1.BIN" from image "image.dsk" and save as "/tmp/file1.bin": $ flextract image.dsk x FILE1.BIN /tmp/file1.bin Extracting the text file "FILE2.TXT" from image "image.dsk" and save as "/tmp/file2.txt": $ flextract image.dsk xt FILE2.TXT /tmp/file2.txt Previewing a file on screen (export to stdout): $ flextract image.dsk xt FILE3.TXT - This a file that is stored on the disk image Adding the "t" option ensures that the file is treated as a FLEX text file. $ 2020 UPDATE: As pointed out by Soren and Alan I have adopted flextract to read FLEX text files more properly now. The "t" option converts the file by stripping out non printable control characters as well as handling the FLEX space compression. FLEX compresses occurrences of consecutive spaces by using a form of run length compression. A 0x09 byte (normally used for tab nowadays) is written to the file followed by another byte determining the number of spaces to be printed.

Executable CMD files

Program files that are executable are a little special. They are stored in FLEX CMD format which is the executable data divided into chunks, each chunk containing its own a load address. This is a flexible solution since it makes it possible to load a single file into different parts of memory before execution. When dealing with the file outside FLEX it's however a bit tricky since CMD files cannot be treated as RAW binary dumps of a program. In order to strip a CMD file of everything but the program code I also had to write the CMD2BIN converter. Source code: cmd2bin.c Compilation and installation under Linux: $ gcc cmd2bin.c -o cmd2bin $ sudo cp cmd2bin /usr/local/bin/ $ cmd2bin Usage: cmd2bin <infile.cmd> <outfile.bin> [start address] [stop address] cmd2bin automatically detects non-continuous parts of the CMD file, which is marked by an extra blank line. $ cmd2bin PROGRAM.CMD PROGRAM.BIN Converting 0000 to FFFF Address: 0100-01C3 Length: 196 Address: 01C4-0287 Length: 196 Address: 0288-034B Length: 196 Address: 034C-040F Length: 196 Address: 0720-07E3 Length: 196 Address: 07E4-08A7 Length: 196 Address: 08A8-096B Length: 196 Address: 096C-0A2F Length: 196 Address: 0A30-0AF3 Length: 196 Address: 0AF4-0BB7 Length: 196 Address: 0BB8-0C7B Length: 196 Address: 0C7C-0D3F Length: 196 Address: 0D40-0E03 Length: 196 Address: 0E04-0E87 Length: 132 $ In the example above PROGRAM.CMD loads into two parts of the memory. Using cmd2bin the program can be exported into separate files for each memory segment. First 0100-040F: $ cmd2bin PROGRAM.CMD PROGRAM_0100.BIN 0100 040F Converting 0100 to 040F Address: 0100-01C3 Length: 196 Address: 01C4-0287 Length: 196 Address: 0288-034B Length: 196 Address: 034C-040F Length: 196 $ Then 0720-0E87: $ cmd2bin PROGRAM.CMD PROGRAM_0720.BIN 0720 0E87 Converting 0720 to 0E87 Address: 0720-07E3 Length: 196 Address: 07E4-08A7 Length: 196 Address: 08A8-096B Length: 196 Address: 096C-0A2F Length: 196 Address: 0A30-0AF3 Length: 196 Address: 0AF4-0BB7 Length: 196 Address: 0BB8-0C7B Length: 196 Address: 0C7C-0D3F Length: 196 Address: 0D40-0E03 Length: 196 Address: 0E04-0E87 Length: 132 $

Summary

Both FLEXTRACT and CMD2BIN has proven very useful to me and hopefully someone else will find them useful as well. They work fine for both FLEX 2 and FLEX 9 images (6800 & 6809). It feels very nice to have a non-windows way of dealing with FLEX images. With this I can only read and not create images but it's a good start!
[show older articles]