Simple IPv4 and IPv6 HTTP Server

When doing hackthebox stuff I often use the SimpleHTTPServer module of python to download scripts and tools from my host system to the client.
Recently I needed an IPv6 http server because IPv4 was blocked. Since I didn’t find a simple way to host files via IPv6 I extent the SimpleHTTPServer module with IPv6 support. While I was at it I also implemented file upload via post.
The latest version can be found on my GitHub page https://github.com/wsoltys/tools
The current version as of this writing is listed below:

#!/usr/bin/python

import sys
import socket
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

class MyHandler(SimpleHTTPRequestHandler):
  def do_GET(self):
    if self.path == '/ip':
      self.send_response(200)
      self.send_header('Content-type', 'text/html')
      self.end_headers()
      self.wfile.write('Your IP address is %s' % self.client_address[0])
      return
    else:
      return SimpleHTTPRequestHandler.do_GET(self)

  def do_POST(self):
      content_length = int(self.headers['Content-Length'])
      body = self.rfile.read(content_length)
      self.send_response(200)
      self.end_headers()
      self.wfile.write('ok')
      filename = self.path[1:]
      with open(filename, 'w') as f:
          f.write(body)
      return

  def do_PUT(self):
      MyHandler.do_POST(self)
      return

class HTTPServerV6(HTTPServer):
  address_family = socket.AF_INET6

def main():
  if len(sys.argv) > 1:
    port = int(sys.argv[1])
  else:
    port = 80  

  server = HTTPServerV6(('::', port), MyHandler)
  print('Serving http on '+server.server_name+' port '+str(server.server_port))
  print('Upload files with: curl -T <file> <server ip>')
  server.serve_forever()

if __name__ == '__main__':
  main()

#!/usr/bin/python

import sys

import socket

from BaseHTTPServer import HTTPServer

from SimpleHTTPServer import SimpleHTTPRequestHandler

class MyHandler(SimpleHTTPRequestHandler):

def do_GET(self):

if self.path == '/ip':

self.send_response(200)

self.send_header('Content-type', 'text/html')

self.end_headers()

self.wfile.write('Your IP address is %s' % self.client_address[0])

return

else:

return SimpleHTTPRequestHandler.do_GET(self)

def do_POST(self):

content_length = int(self.headers['Content-Length'])

body = self.rfile.read(content_length)

self.send_response(200)

self.end_headers()

self.wfile.write('ok')

filename = self.path[1:]

with open(filename, 'w') as f:

f.write(body)

return

def do_PUT(self):

MyHandler.do_POST(self)

return

class HTTPServerV6(HTTPServer):

address_family = socket.AF_INET6

def main():

if len(sys.argv) > 1:

port = int(sys.argv[1])

else:

port = 80

server = HTTPServerV6(('::', port), MyHandler)

print('Serving http on '+server.server_name+' port '+str(server.server_port))

print('Upload files with: curl -T <file> <server ip>')

server.serve_forever()

if __name__ == '__main__':

main()

Reading FAT32 inside a FPGA core

Current FPGA boards have support for a sdcard but not all cores support it. There’re some classic cores (like Acorn Atom or the Spectrum) which do it with an extension ROM. But often the sdcard needs to be in a special format not readable by a PC which is also a problem for the MiST because it needs to load the core from a FAT filesystem and currently its not supported to change the sd card during runtime.
Another solution is to instantiate a second CPU which does the FAT reading and injects the code into RAM addressable by the core. But this comes with extra code because its like a second machine with own bios inside the fpga. Examples for that are the MSX and PC Engine port for the MiST.
Wouldn’t it be nice to keep requirements small and read a FAT filesystem directly in the core?

The FPGAmstrad core from Renaud Hélias raised the idea but I wanted to code my own module for learning purpose. But where to start? First I needed a library to easily read the sdcard block by block. I found code from XESS which was a good start (https://github.com/xesscorp/VHDL_Lib/blob/master/SDCard.vhd).
Next was a comprehend guide to understand FAT. A good summary with the most important bits are Paul’s 8051 Code Library.

The code is written around a FSM (finite state machine). I started with reading the first sector of the sdcard to determine the first partition. Before reading the first sector of the first partition the code checks if its a valid FAT filesystem and what type it is. The current code supports only FAT32 and one partition. But this should cope with most situations as today you can only buy SDHC cards but even SD cards can be formatted with FAT32.

when FAT_CHECK =>
  if block_mem(SIGNATURE_POSITION) = x"55" and block_mem(SIGNATURE_POSITION+1) = x"AA" then
    state <= PARTITION_TYPE;
  end if;
when PARTITION_TYPE =>
  -- we only support fat32 here
  case block_mem(PARTITION1_TYPECODE_LOCATION) is
    when x"0B" => valid_partition <= '1';
      state <= CHECK_LBA;
    when x"0C" => valid_partition <= '1';
      state <= CHECK_LBA;
    when x"00" => valid_partition <= '0';
      state <= CHECK_LBA;
    when others => state <= ERROR;
  end case;

when FAT_CHECK =>

if block_mem(SIGNATURE_POSITION) = x"55" and block_mem(SIGNATURE_POSITION+1) = x"AA" then

state <= PARTITION_TYPE;

end if;

when PARTITION_TYPE =>

-- we only support fat32 here

case block_mem(PARTITION1_TYPECODE_LOCATION) is

when x"0B" => valid_partition <= '1';

state <= CHECK_LBA;

when x"0C" => valid_partition <= '1';

state <= CHECK_LBA;

when x"00" => valid_partition <= '0';

state <= CHECK_LBA;

when others => state <= ERROR;

end case;

Once we have the LBA of the first partition we read the first sector and derive some important values from there:

when CHECK_PARTITION =>
  -- check for a valid partition
  if block_mem(11) = x"00" and block_mem(12) = x"02" and block_mem(16) = x"02" and block_mem(SIGNATURE_POSITION) = x"55" and block_mem(SIGNATURE_POSITION+1) = x"AA" then
    sectors_per_cluster <= block_mem(13);
    fat_begin_lba <= resize(lba_begin + (block_mem(15) & block_mem(14)), fat_begin_lba'length);
    cluster_begin_lba <= resize(lba_begin + (block_mem(15) & block_mem(14)) + (block_mem(16)*(block_mem(39) & block_mem(38) & block_mem(37) & block_mem(36))), cluster_begin_lba'length);
    root_dir_first_cluster <= block_mem(47) & block_mem(46) & block_mem(45) & block_mem(44);
    -- compute block address: addr = (cluster_begin_lba + (root_dir_first_cluster-2)*sectors_per_cluster)*FAT_SECTOR_SIZE
    block_addr <= resize((lba_begin + (block_mem(15) & block_mem(14)) + (block_mem(16)*(block_mem(39) & block_mem(38) & block_mem(37) & block_mem(36)))
                      + ((block_mem(47) & block_mem(46) & block_mem(45) & block_mem(44))-2)
                        * block_mem(13))*sd_factor, block_addr'length);
    state <= READ_BLOCK1;
    return_state <= READ_DIR1;
  else
    state <= ERROR;
  end if;

when CHECK_PARTITION =>

-- check for a valid partition

if block_mem(11) = x"00" and block_mem(12) = x"02" and block_mem(16) = x"02" and block_mem(SIGNATURE_POSITION) = x"55" and block_mem(SIGNATURE_POSITION+1) = x"AA" then

sectors_per_cluster <= block_mem(13);

fat_begin_lba <= resize(lba_begin + (block_mem(15) & block_mem(14)), fat_begin_lba'length);

cluster_begin_lba <= resize(lba_begin + (block_mem(15) & block_mem(14)) + (block_mem(16)*(block_mem(39) & block_mem(38) & block_mem(37) & block_mem(36))), cluster_begin_lba'length);

root_dir_first_cluster <= block_mem(47) & block_mem(46) & block_mem(45) & block_mem(44);

-- compute block address: addr = (cluster_begin_lba + (root_dir_first_cluster-2)*sectors_per_cluster)*FAT_SECTOR_SIZE

block_addr <= resize((lba_begin + (block_mem(15) & block_mem(14)) + (block_mem(16)*(block_mem(39) & block_mem(38) & block_mem(37) & block_mem(36)))

+ ((block_mem(47) & block_mem(46) & block_mem(45) & block_mem(44))-2)

* block_mem(13))*sd_factor, block_addr'length);

state <= READ_BLOCK1;

return_state <= READ_DIR1;

else

state <= ERROR;

end if;

sd_factor is 512 for SD cards (byte addressing) and 1 for SDHC cards (block addressing).

With those values we know where to start reading the root directory and can search for the file we want to load. Subdirectories are also located here but the current code only supports reading from the root directory and from small directories as FAT reading for dirs isn’t supported yet 🙂
For FAT32 the directory entries are 32 bytes long and we jump from entry to entry until we’ve found our file.

when COMPARE1 =>
  if block_mem(sector_cnt + entry_cnt) = filename(87-(entry_cnt*8) downto 80-(entry_cnt*8)) then
    if entry_cnt = 10 then
      -- filename found
      file_size <= block_mem(sector_cnt+31) & block_mem(sector_cnt+30) & block_mem(sector_cnt+29) & block_mem(sector_cnt+28);
      next_cluster <= block_mem(sector_cnt+21) & block_mem(sector_cnt+20) & block_mem(sector_cnt+27) & block_mem(sector_cnt+26);
      ram_addr_o <= unsigned(file_ram_a);
      state <= READ_FILE1;
    else
      entry_cnt <= entry_cnt + 1;
      state <= READ_DIR1;
    end if;
  else
    -- next entry
    sector_cnt <= sector_cnt + 32;
    entry_cnt <= 0;
    if block_mem(sector_cnt) = x"00" then
      -- file not found
      state <= ERROR;
    else
      state <= READ_DIR1;
    end if;
  end if;

when COMPARE1 =>

if block_mem(sector_cnt + entry_cnt) = filename(87-(entry_cnt*8) downto 80-(entry_cnt*8)) then

if entry_cnt = 10 then

-- filename found

file_size <= block_mem(sector_cnt+31) & block_mem(sector_cnt+30) & block_mem(sector_cnt+29) & block_mem(sector_cnt+28);

next_cluster <= block_mem(sector_cnt+21) & block_mem(sector_cnt+20) & block_mem(sector_cnt+27) & block_mem(sector_cnt+26);

ram_addr_o <= unsigned(file_ram_a);

state <= READ_FILE1;

else

entry_cnt <= entry_cnt + 1;

state <= READ_DIR1;

end if;

else

-- next entry

sector_cnt <= sector_cnt + 32;

entry_cnt <= 0;

if block_mem(sector_cnt) = x"00" then

-- file not found

state <= ERROR;

else

state <= READ_DIR1;

end if;

Once found the entry contains the file size and the starting cluster. From there we can read the file sector by sector and cluster by cluster. For my 4GB SDHC card the cluster size was 4KB (8 sectors * 512byte). After reading a full cluster we need to find the next cluster via the FAT and so on.

The current code is still work in progress as it contains a lot of debug stuff and is missing some important bits:
– Reading longer directories via FAT
– Optimize code as it currently takes up to 12.000 LE’s on a Cyclone III (and I have no idea why)
– Support subdirectories
– Support writing would be nice but I don’t think that I get to it soon

At least it works which was a good start for me 🙂
The code is in my github repo as usual if you want to give it a try.
https://github.com/wsoltys/mist-cores/tree/master/misc/sdcard

Disclaimer: The code was written for learning purpose and might be highly inefficient. I’m open for any suggestions to optimize the code.

How to generate a VGA signal with a FPGA

Update 2015/03/08:
Added DE1 board files and a small bug fix. Fixed the repo link below to point to the right revision.

During my efforts to learn more about fpga’s I found a nice tutorial about VGA in VHDL on the DE1 board on youtube.
I was surprised how easy it is to do it when you got it explained 🙂

In short, you draw the pixel line by line on the screen and every line contains an active video phase in which the rgb pixels are sent and a blanking region which is divided into the front porch, sync pulse and back porch (see image below).

vga line

In the youtube tutorial its told that it starts with the blanking region followed by the active video phase. Probably it doesn’t matter but I did it in the same way (please comment if you know whats right and if it makes any difference).
It turned out that by starting with the visible area first the bending which was visible on the horizontal line is gone. I’ve updated the code accordingly.
The pixel clock and the pixel count of front porch, sync pulse, back porch and the active video phase is defined and can be seen here:
http://www-mtl.mit.edu/Courses/6.111/labkit/vga.shtml
http://tinyvga.com/vga-timing

I chose the resolution 640×480 with a pixel clock of 25.175 MHz. Let’s have a look at the code:

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

-- http://tinyvga.com/vga-timing/640x480@60Hz

ENTITY VGASYNC IS
  PORT(
    CLK: IN STD_LOGIC; -- pixel clock 25.175 MHz
    HSYNC,VSYNC: OUT STD_LOGIC;
    R,G,B : OUT STD_LOGIC_VECTOR(5 downto 0)
  );
END VGASYNC;

ARCHITECTURE MAIN of VGASYNC IS

-- Horizontal timing (line)
constant hva : integer := 640; -- Visible area
constant hfp : integer :=  16; -- Front porch
constant hsp : integer :=  96; -- Sync pulse
constant hbp : integer :=  48; -- Back porch

-- Vertical timing (frame)
constant vva : integer := 480; -- Visible area
constant vfp : integer :=  10; -- Front porch
constant vsp : integer :=   2; -- Sync pulse
constant vbp : integer :=  32; -- Back porch

signal HPOS: integer range 0 to 800:=0;
signal VPOS: integer range 0 to 525:=0;

library ieee;

use ieee.std_logic_1164.all;

use ieee.numeric_std.all;

-- http://tinyvga.com/vga-timing/640x480@60Hz

ENTITY VGASYNC IS

PORT(

CLK: IN STD_LOGIC; -- pixel clock 25.175 MHz

HSYNC,VSYNC: OUT STD_LOGIC;

R,G,B : OUT STD_LOGIC_VECTOR(5 downto 0)

);

END VGASYNC;

ARCHITECTURE MAIN of VGASYNC IS

-- Horizontal timing (line)

constant hva : integer := 640; -- Visible area

constant hfp : integer := 16; -- Front porch

constant hsp : integer := 96; -- Sync pulse

constant hbp : integer := 48; -- Back porch

-- Vertical timing (frame)

constant vva : integer := 480; -- Visible area

constant vfp : integer := 10; -- Front porch

constant vsp : integer := 2; -- Sync pulse

constant vbp : integer := 32; -- Back porch

signal HPOS: integer range 0 to 800:=0;

signal VPOS: integer range 0 to 525:=0;

Our vga entity will get the pixel clock as input and outputs the horizontal and vertical sync pulse together with the rgb values (6 Bit for the MiST).
Below architecture are some constants which match the given values of our resolution. HPOS and VPOS are the horizontal and vertical counters which end positions are the sums of the above values.

process(CLK)
begin
if rising_edge(CLK) then
  
  if HPOS < (hva+hfp+hsp+hbp) then
    HPOS <= HPOS + 1;
  else
    HPOS <= 0;
    if VPOS < (vva+vfp+vsp+vbp) then
      VPOS <= VPOS + 1;
    else
      VPOS <= 0;
    end if;
  end if;

process(CLK)

begin

if rising_edge(CLK) then

if HPOS < (hva+hfp+hsp+hbp) then

HPOS <= HPOS + 1;

else

HPOS <= 0;

if VPOS < (vva+vfp+vsp+vbp) then

VPOS <= VPOS + 1;

else

VPOS <= 0;

end if;

With the pixel clock we change our counters. As long as HPOS is below 800 we increase it by one. If its higher we’re in the next line and set HPOS back to zero (begin of the next line). Then we check if VPOS is below 524 and if yes increase it by one and if not set it to zero (begin of the next frame).

  if HPOS > (hva+hfp) and HPOS < (hva+hfp+hsp) then
    HSYNC <= '0';
  else
    HSYNC <= '1';
  end if;
  
  if VPOS > (vva+vfp) and VPOS < (vva+vfp+vsp) then
    VSYNC <= '0';
  else
    VSYNC <= '1';
  end if;

if HPOS > (hva+hfp) and HPOS < (hva+hfp+hsp) then

HSYNC <= '0';

else

HSYNC <= '1';

end if;

if VPOS > (vva+vfp) and VPOS < (vva+vfp+vsp) then

VSYNC <= '0';

else

VSYNC <= '1';

end if;

From the figure above we see that HSYNC is only zero during the sync pulse. The same is valid for VSYNC. And thats exactly what the code above does.

  if (HPOS > hva) or (VPOS > vva) then
    R<=(OTHERS=>'0');
    G<=(OTHERS=>'0');
    B<=(OTHERS=>'0');
  else
  
    -- Blue background
    R<=(OTHERS=>'0');
    G<=(OTHERS=>'0');
    B<=(OTHERS=>'1');
    
    -- White cross-hair
    if(HPOS > 475 and HPOS < 485) or (VPOS > 280 and VPOS < 290) then
      R<=(OTHERS=>'1');
      G<=(OTHERS=>'1');
      B<=(OTHERS=>'1');
    end if;   
     
  end if;
end if;
end process;

END MAIN;

if (HPOS > hva) or (VPOS > vva) then

R<=(OTHERS=>'0');

G<=(OTHERS=>'0');

B<=(OTHERS=>'0');

else

-- Blue background

R<=(OTHERS=>'0');

G<=(OTHERS=>'0');

B<=(OTHERS=>'1');

-- White cross-hair

if(HPOS > 475 and HPOS < 485) or (VPOS > 280 and VPOS < 290) then

R<=(OTHERS=>'1');

G<=(OTHERS=>'1');

B<=(OTHERS=>'1');

end if;

end process;

END MAIN;

The code above sets our pixels. The first section sets the rgb values to zero when we aren’t in the active video phase. Then the background is set to blue and “overdrawn” by white when the HPOS and VPOS matches certain values which will show a horizontal (HPOS) and a vertical (VPOS) white stripe in the middle of the screen.

The full source code of this example together with the Quartus project files for the MiST can be found here:
https://github.com/wsoltys/mist-cores/tree/77fea2dc1e3986daa9138dbb2fa91fd8267c3c9e/misc/vga

How to convert binary files to Altera readable input

Just as a reminder for me.
From time to time I need to convert a binary ROM file (bios or cartridge data) to a format which the Altera Quartus Suite can understand. It can be in hex but I chose the Memory Initialization File (MIF) format by Altera.

srec_cat <input file> -binary -o <output file>.mif -mif

1	srec_cat <input file> -binary -o <output file>.mif -mif

srec_cat: http://sourceforge.net/projects/srecord/files/srecord-win32/

Compiling the minimig core for the MIST fpga board on windows

Compiling the Atari ST core for the MIST is pretty straight forward like described here. For the minimig core some prerequisites are needed which I’ll describe below.
The amiga_boot.v file is missing and needs to be compiled before using the Altera Web Edition to compile the core.
I was lazy and took the mingw build environment used for XBMC (https://github.com/xbmc/xbmc/tree/master/project/BuildDependencies). I used it to compile vasm which is needed to compile amiga_boot.

Get the sources from here: http://sun.hasenbraten.de/vasm
Unzip it to vasm
Go into vasm and type
make CPU=m68k SYNTAX=mot

I copied the binary vasm68k_mot.exe to a bin directory in the mingw env. Since the xxd binary was missing as well I download the msys-vim package from sourceforge and installed it in the mingw env as well.
When I got some time I’ll create a stripped down mingw/msys env with the binaries needed to compile the firmware and amiga_boot.
Once done we need python for windows in path. With some smaller changes to the python scripts and the makefile I was able to compile the amiga_boot files. Once finished I’ll submit the changes to the MIST main repo.
Meanwhile you can use the files from here: https://github.com/wsoltys/mist-board/tree/master/cores/minimig/fw/amiga_boot/bin

Copy those files to the same directory structure of the MIST main repo clone and open minimig_mist.qpf with the Altera Web Edition. In the Web Edition you need to enable talkback in order to use the Signaltap which is needed to compile the minimig core.

Go to Options
Internet Connectivity
Talkback options and enable them

Now you can start the compilation and after some minutes its done 🙂

WiSo's collector blog

things I would like to remember

Coding

Simple IPv4 and IPv6 HTTP Server

Reading FAT32 inside a FPGA core

How to generate a VGA signal with a FPGA

How to convert binary files to Altera readable input

Compiling the minimig core for the MIST fpga board on windows