Working with Files in NodeMCU on the ESP8266

By December 15, 2015ESP8266, IoT

What is SPIFFS?

One of the coolest things about the NodeMCU firmware is that it includes a SPI Flash File System (SPIFFS) module. This means that data and scripts can be written to and read from the flash memory using the concept of files instead programming against raw memory locations.

Flash memory allow random access reads and writes. But, by nature, flash memory only allows erasure for entire blocks of memory (i.e., 64 kB) at a time. When erased, a NOR flash memory address will contain all ones, and writing to NOR flash memory will set all of the zeros in the byte being written, but cannot set the ones. This means that writing 0b10101010 to an address containing 0b00001111 will result in 0b00001010, since the first nibble in that address was already zeroed out by a previous write. The only way to put ones back into an address is to clear the entire block again.

Oh, and one more thing about Flash memory: it can only be erased so many times (100,000 to 1,000,000 typical) before failure. So, if you have a program that logs to the same block of memory and performs a lot of erasures without consideration for this, then it’s possible to ruin your ESP8266 module in a short time.

So, given the complexity of working with flash memory, heck yeah I want an abstraction layer over that!

SPIFFS manages this complexity, and even takes advantage of NOR flash memory’s ability to always being able to write zeros. To create a filesystem, it partitions the available memory into pages that are small enough to be able to cache in RAM (256 bytes per page is the default). Some pages contain index information, and the other pages contain file contents.

SPIFFS attempts to optimize writes in the interest of conserving flash memory and limiting the number of erasures. Performing small writes (e.g., writing one byte at a time within a loop) will not immediately be written to flash, but rather to a buffer in RAM. Once that buffer fills up, it will then be written to the flash and a new page will be started. If data in the middle of a page needs to be altered, then the entire page is marked as deleted in flash and the changes are copied into a new page. When necessary to erase a block in order to reclaim “disk space”, any active pages are copied into a different block first.

Working with the “file” Module

Within NodeMCU, the file module is used by Lua scripts to interact with the SPIFFS filesystem. This permits files to be listed, renamed, and removed at the file system level, and individually, files can be opened for random access reads and writes. The entire file system can also be formatted, which erases all blocks of data.

Let’s create a file:

When you open a file for writing that doesn’t already exist, then NodeMCU will create it for you. There can only be one file open at a time, so there is no need to maintain a file handle between operations.

SPIFFS doesn’t have the concept of folders, but accepts arbitrary characters in the filename, so you can create pseudo-folders just by adding slashes within the file’s name. Note that the maximum filename length is 32 characters, and extensions on the end of a filename do not have any special meaning (except for lua-specific files, like .lua and .lc).

The .writeline() function will append a newline character (\n) to the end of the string that is provided. To write single bytes at a time, use the .write() function instead.

And, finally, closing the file will flush the buffer and cause the flash memory to be written.

We can list what’s on the filesystem (a.k.a., perform a “dir” or “ls”) with a bit of code:

A bit more complex than just typing “dir”, yes. file.list() will return a Lua table that uses the file name as a key, and the file length as the value. So, to iterate this dictionary from Lua, you need to use a for-loop in conjunction with the pairs() function (which provides the iterator over the key-value pairs). And, in case you’re new to Lua, the string concatenation operator is two periods (..).

Reading the file is similar to writing:

The default mode for file.open() is “r”, which is read-only. It’s usually a good idea to check the return value from file.open() to ensure that the file exists (return value = true) before you try to do something:

The file can be renamed:

And finally, the file can be deleted:

Documentation for the file module can be found in the NodeMCU Documentation.

Uploading Files to the ESP8266

One downside of the NodeMCU ecosystem is that there is no official IDE for script developers to use. The ones that do exist lack a certain sense of polish. However, they provide mechanisms to upload files from your machine onto the ESP8266’s file system, so it’s better than nothing.

The two programs that I’ve used so far include LuaLoader for Windows and ESPlorer (written in Java, so multi-platform). ESPlorer appears to be more polished of the two with a multiple docked window interface, but personally, I don’t run Java on my main development machine, so I find myself defaulting to LuaLoader (Note: I do use ESPlorer on a Raspberry Pi 2 running Raspian, and it’s usable, but really benefits from having a faster processor and more memory available on a larger machine).

LuaLoader assumes the role of a terminal. It exposes common NodeMCU functionality as macros that can be executed via button clicks.

lualoader

Let’s upload a file! First, create a text file on your computer named “hello.lua”. Edit it so that its contents are:

LuaLoader suffers from a “chicken-and-egg” problem when it comes to file uploads. Since NodeMCU doesn’t provide a file upload API, a small Lua program is used by LuaLoader to read in bytes from the UART and write them to the filesystem. But, if that program does not already exist on the ESP8266 device, then LuaLoader will enter into an endless loop of fail when you try to upload a file… including the uploader program itself!

The solution is to first “type” the file contents of LLbin.lua to the device. This is the same thing as if you had typed the program yourself into the REPL, and creates a global function called LLbin() that LuaLoader will invoke for its file uploads.

To “type” the file, click the “Type” button on the bottom of the screen, and then browse for “LLbin.lua” inside of the LuaLoader folder. You should see the contents of the program appearing on the screen.

Next, so that you don’t have to do this every time that you want to upload a file, you should upload LLbin.lua itself so that it’s on the ESP8266’s file system (this is a one-time operation). Click the “Upload File” button on the right, and browse again for “LLbin.lua”, and then click “Open”. You should see something like the following:

Okay, now we’re ready to upload our own script. In LuaLoader, click the “Upload File” button on the right. Browse for your “hello.lua” file, and then click “Open”.

Once the file has been uploaded, click the “file.list” button on the lower right. (This is nicer than having to type out that line of Lua code, right?!). You should see two files now:

To run a Lua file, use the “dofile()” function:

If you are writing a program that needs to run every time that the ESP8266 resets (as you would in an IoT scenario), then just name that file “init.lua”:

Now reset the device (either using the RESET button on the device, or click the “Restart” button on the right of the LuaLoader screen). Instead of seeing a NodeMCU warning stating that it could not find init.lua, you will see “Hello World!” printed after the banner text:

A word of caution, though: if bad code executes in init.lua that causes the ESP8266 to Panic and reset, then you could enter into an endless panic-reset loop that will be impossible to stop without re-flashing the firmware. Therefore, it’s always a good idea to insert a delay at the beginning of the init.lua code to give you time to execute file.remove(“init.lua”) from a terminal. Another common approach of preventing panic-reset loops is to check a GPIO pin that you can physically control, and execute the code only if it is a default value (i.e., maybe a momentary switch will connect that pin to GND, otherwise it floats high). If you find yourself in a panic-reset loop, then set the pin value to the non-default state and the code will not execute, giving you the chance to upload a new init.lua with bug fixes.

So that’s the basics of the NodeMCU file system! This feature allows NodeMCU to remain as a general-purpose firmware that you can use in your own Thing, yet you are still able to customize its behavior by means of creating Lua scripts that survive device resets. Configuration and assets can be stored on the ESP8266 itself, and data can be logged to the file system for storage as needed (i.e., maybe in the event that the network is not available).

The following two tabs change content below.
  • Pingback: Working with Files in NodeMCU on the ESP8266 | Dinesh Ram Kali.()

  • Pingback: Working with Files in NodeMCU on the ESP8266 | TRIFORCE STUDIO()

  • Arantis Fx

    HI
    thanks a lot for your post
    I have the error
    inary upload: C:UsersgunteGoogle DriveNodeMcuLuaLoaderversion 0.91LLbin.lua
    stdin:1: attempt to call global ‘LLbin’ (a nil value)
    > > dofile(“LLbin.lua”)
    cannot open LLbin.lua
    > uart.setup(0,9600,8,0,1,0)

    have you an idea
    thanks

  • flagtrax

    Hi I read with great interest. Nicely done, and helps me greatly in gaining more understanding of the 8266. I’ve vacillated between NodeMCU and the Arduino IDE to create a wifi based controller using 5 GPIO pins using the 8266 as a server returning an HTML file to the calling device browser. Both worked in their simple forms, but when I tried make the HTML buttons more sophisticated I ended up using inline styling to the HTML elements. This only worked to a point using ESPlorer; but worked fully when loading the same code through the Arduino IDE. When Esplorer “compiled” the code it reported errors after the code grew to provide all the wanted attributes to the button. I was defining button size, font size, and background color and I’d get an error If I took out any one of the settings and tried again the statement would not error. It didn’t matter which one. It was as if there was some type of buffer overflow or something in the ESPlorer IDE. I have completed what I needed to do at this stage, using Arduino IDE, but I’d like to make the Web page a bit more sophisticated to the point I don’t think inline styling is the proper way to go. I’d like to be able to at some point upload corresponding HTML and CSS files to be delivered to the calling wifi device. You state “There can only be one file open at a time, ………………..” Does this mean that (1) since init.lua is running that another file can’t be opened, and (2) Since CSS and HTML coding would be in two different files how can that be accessed to present the proper page formatting? Thanks for any input.

    • What Jason means in saying that only one file can be open, is that your Lua code can only have one file handle open and it must be closed before opening another. It does not mean that you can’t access files while the runtime is executing init.lua—this would make it pretty impossible to ever use files in your code 🙂

      In the example file.open("hello/world.txt") the file “object” is stateful, it’s now operating on hello/world.txt and you need to do something with that file’s contents and then call file.close() before you can try to work with another one. You could read the contents as a string and assign them to a variable, then close the file and read another one to another variable. Just be mindful that memory is precious on these devices and big strings loaded eagerly rather than iterated efficiently can be costly.

      • flagtrax

        Thanks, that makes sense ?