Friday, February 16, 2018

ESP Voice Control with ESP-Basic

For an index to all of my stories click here.

Voice control is the next big thing in (home) automation. Just look at all the companies wanting a piece of the cake: Amazon with Alexa, Apple with Siri and Google with Home Assistant. And indeed when you start playing with voice recognition and synthesizes you realise how much fun it is. Do I believe in it ?? I am not quite sure yet. There are some pitfalls, however I do believe that the large companies will be pushing it into our homes sooner or later.

I already wrote a major story on Google's Home Assistant running on a Raspberry Pi and controlling an ESP-8266.

And there are two stories on this weblog that also use voice control but take a different approach. They do not use the Home Assistant but do use voice recognition through a dedicated Android APP. The first project used Bluetooth to communicate with an Arduino and was written in 2016. The second project used Wifi to communicate with an ESP8266

In these previous entries I showed you how to control an ESP8266 by an Android App instead of the usual web/page control.

For those who did not read these stories here is a brief explanation.
Normally you control an ESP8266 by a webpage with buttons, sliders etc etc. This can rise a problem as anyone in your network can control the ESP through that webpage. You can build in some safety measures but that involves a lot of work.
When you control an ESP/8266 with an Android dedicated App only those who have the App can control the ESP.

Now this project is not exactly a worthy opponent of Google Assist but it does the job it was designed for remarkably well.

The version I am going to present you today is even more remarkable as it uses a general purpose Voice App which I specailly devellopped and a very easy to understand program in ESP-Basic.

The difference in approach.

In the previous version where I commanded the ESP by voice I showed you how to make an app that reacted to several fixed commands. You can re-read that story here: https://lucstechblog.blogspot.nl/2018/02/wifi-voice-commander.html This works but is however not very flexible. To alter the command you must alter the App in Mit's App Inventor which is fairly easy done however time consuming. Next to that you will need to alter the ESP program also if you add or remove commands. And that involves re-writing the C++ program, compiling it again and transferring it to the ESP-8266.

In this story I am going to show you how to make a general purpose App for Android that only translates the speech to text which is then send to the ESP8266 to be analysed.
This way you only need to alter the ESP-Basic program if you want to alter commands and that is easily done.

Mit App Inventor and ESP-Basic.

This is not going to be a tutorial on how to use App-Inventor. There is ample doncumentation around on that. Mit's own website is full of tutorials and sample programs. So please check that first if you are not familiar with this easy to use Android Devellopment platform.
This is also no tutorial on how to work with ESP-Basic. That has been covered  in a previous story on this web-log that you can read here.

https://lucstechblog.blogspot.nl/2017/03/back-to-basic-basic-language-on-esp8266.html

Working with Mit's App Inventor really is very easy and you will be designing and programming Android App's in notime. Communication with the ESP using Wifi or an Arduino over Bluetooth is a piece of cake and therefoer I urge you to delve into this as I will publish some more projects using this tool.

Off course there is a link to both the App and the source code at the end of this story.

The Designer screen

Start with downloading the Wifirelay.aia source code to your computer. Then point your browser to http://appinventor.mit.edu/explore/
On the right side of the screen click on the orange square that says "Create Apps"
Now you can log in with your Google credentials. If you do not have a Google account make sure you get one first.
In the projects menu on the top of your screen choose -- import project(.aia) from my computer.



When done your screen will look the same as above.

On the right side of the screen you will find a column with all the components that are used in your App. Clicking a component will highlight it on the example screen and in the last column on the right side of the screen you will be able to alter the settings for that component.

As you can see there are a few labels that do nothing more as putting text on your screen. Then there is a textbox that allows you, when running the App, to fill in the IP adress of the ESP8266 that is going to be connected.
Next there is a button called "Voice_input" whick I gave an image of a microphone.
And below that there is an invisible label. That label will get visible after you spoke your command and will be filled with the text you spoke. This makes sure you have a visible check if the command was pronounced well and recognised well.
The last two components are the Web1, Speechrecogniser1 and TextToSpeech1 component. They are very important and working in the back-ground. The Web component takes care of the communiaction between the ESP8266 and the App. The Speechrecogniser1 will translate your spoken commands into mwritten text and the TextToSpeech1 component is going to be used to have the App speak out loud when the function has completed.



There is one thing I really want to Emphasize.
Look at the TextToSpeech1 component carefully. If you leave it at its default values the App will speak out any text in the language you have set as your default language in your phone. If that is English all is well. However my phone is set default to Dutch. Therefore English texts will not be recognised well by the Speechrecogniser and the confirmation text will be spoken out awkward. So if you are going to use another language as English make sure you set this to your preferred language.

When all is set switch on the right (top) side of the screen to the blocks section wher the actual program will be written.

The Blocks

Well the program almost speaks for itself except for one small detail which I will get into in detail.
 

First a global variable called input1 is initialised.
Then the program just waits till you press the button called "Voice_input"

Next comes the main part.

First the result from the Speechrecogniser is put in the global variable "global_input1"
The next 2 lines in that block replace any spaces in your spoken text by a "."
The reason for this is that the web communication, for some reason or another, will only send the first word in a sentence.
Now we do not want to send one word commands. We want to send complete sentenses like "living room ceiling light on".
If you speak out this command the web component will only send the word "living".
So the program lines change this in "living.room.ceiling.light.on"

In the next step, which is the ESP-Basic program, we will filter the dots out again.

The following sequence of lines actually sends the command over the net.
The command is build up by the word "http://" followed by the IP_Adress the word "/msg?saywhat=" and the command you spoke in all lower case letters.

This sums up to the following.
If your command would be "living room ceiling light on" the command send over the network would be: http://xxxx.xxxx.xxxx.xxxx/msg?saywhat=living.room.ceiling.light.on

You can alter "saywhat" in another word, but then you will have to alter this later on in the ESP program to in exact the same word. This way you can build some security into your program.

And the last block gives an audible feedback that confirms that your command has been send. I used the phrase "yes master" but you can alter it to your own liking. I like a touch of slavery in my electronics.


In the video at the bottom of this story you can see that I altered the App a bit so it repeats the command I gave instaed of saying "yes master" I also edited the Basic program a bit to incorporate a command that puts all lights off.

In the previous entries in this series the program was written in C++ (Arduino code). Nothing wrong with that however this time I am going to write the program in ESP-Basic which gives us a lot of ease and flexibility.

The Basic program



msgbranch [mybranch]

textbox command

wait

[mybranch]
command = msgget("saywhat") 
command = replace(command , "." , " ")
print command
wait

Let us start with a test.

This small program starts when the information is received over wifi. The -- msgbranch -- command is executed and at the program jumps to the mybranch subroutine.

In the mybranch subroutine the first thing that happens is a test if the received information starts with "saywhat" this is the trigger that actually starts the code. If  "saywhat" is not received nothing will happen.

Next step is that any received dots are replaced by spaces again. This way the sentence in our example "living.room.ceiling.light.on" is replaced by "living room ceiling light on". Remember we have to do that because the Wifi communication does not treat spaces well.

The program prints the received command and also places it in the textbox.

This way we have a check if the spoken commands are recieved well and not misinterpreted.

For this project I have used the commands:

living room on / off
garage door open / close
garage light on /off
bathroom light on /off
living room ceiling light on / off
water pump on / off


Recognise these commands ??
Yep they are the same commands I used in my story about commanding the ESP8266 by Google Home Assistant which you can read here: https://lucstechblog.blogspot.nl/2017/05/google-home-and-esp8266.html
In fact is even exact the same Basic program.

So let us have a closer look at the complete program.



'-----------------------------
'init
'-----------------------------
io(po,d1,0)
io(po,d2,0)
io(po,d3,0)
io(po,d5,0)
io(po,d6,0)
io(po,d7,0)

msgbranch [mybranch]

textbox command

wait

[mybranch]

command = msgget("saywhat") 
command = replace(command , "." , " ")
print command
if command = "living room on" then
io(po,d1,1)
endif
if command = "living room off" then
io(po,d1,0)
endif

if command = "garage door open" then
io(po,d2,1)
endif
if command = "garage door close" then
io(po,d2,0)
endif

if command = "garage light on" then
io(po,d3,1)
endif
if command = "garage light off" then
io(po,d3,0)
endif

if command = "bathroom light on" then
io(po,d5,1)
endif
if command = "bathroom light off" then
io(po,d5,0)
endif

if command = "living room ceiling light on" then
io(po,d6,1)
endif
if command = "living room ceiling light off" then
io(po,d6,0)
endif

if command = "water pump on" then
io(po,d7,1)
endif
if command = "water pump off" then
io(po,d7,0)
endif

wait

After the init part where all IO ports are set as output and put off the program starts a MESSAGE BRANCH.
It waits till it gets a command from the internet and displays that command in a textbox on your webpage. This makes it easy to determine if your spoken commands are in the right way interpreted by the App and well transmitted over wifi.
If certain commands are always faulty interpreted you can use the Basic test program (from above) and use a different spoken command.



[mybranch]

command = msgget("saywhat") 
command = replace(command , "." , " ")
print command
if command = "living room on" then
io(po,d1,1)
endif
if command = "living room off" then
io(po,d1,0)
endif

When a command has been received from the App the program jumps to [mybranch]

The first thing that happens is to see if  -- saywhat -- has been received. You can alter that in the program to any other word you like. However remember to change that phrase in the APP also to exact the same word. If -- saywhat -- (or your own choice) is not received the program will not do anything.

The next line is a bit tricky:

command = replace(command , "." , " ")

As said above web communication has a bit of a problem with spaces. Therefore we altered thye spaces in the APP in dots (".") this line changes them back in spaces.

To see if everything went well the received command is printed on your webpage by the print command.

The next lines decide which command has been received and accordingly put the accompaning I/O port on or off.

As you can see this far more easier as the C++ (Arduino code) version and much easier to edit. Better still: the Basic program can be altered over the Air. When working with C++ you will always have to attach the ESP to a computer. So all in all this is more convenient. And remember in this version you only need to alter the Basic program if you need to change commands. The APP stays the same. You just give other voice commands.

That is all folks !!

The hardware





This is as you can see very straightforward just a bunch of leds connected to a delimiter resistor of 220 Ohm and attached to the I/O ports of the ESP8266.


Getting the IP adress

We need to know the IP adress of the ESP8266 to be able to send commands to it.

So open your web-browser and poit it to the IP adress of your router. Most likely that will be 192.168.1.1 but consult your routers manual for this.



So delve into your router and look at your connected devices. Mine looks like above.

The lower icons are the wired apparatus and as you can see it is my computer (Nescio), printerserver (actually a Raspberry) called Epson as it is connected to my Epson printer, my Buffalo Nas, my Domoticz domotica system and a raspberry. The last one is my DIY Google Home.

The top line icons are my wireless apparatus. There you can see a Thermometer (an ESP with a DS-18b20) and a device called 165. That last one is the ESP this all is about. I named it 165 as this is its IP adress and easily remembered. So in my case the local IP adress is 192.168.1.165

I assume you are going to to use this from within your local network. If you are going to use this from other places as your home you will need to make your ESP8266 accessible from the outside world. You can do this by opening a gate (port) in your router and use port-forwarding.

There is a detailed description on this in my story about attaching an ESP8266 to Google Home which you can read here http://lucstechblog.blogspot.nl/2017/05/google-home-and-esp8266.html (about halfway down the story).

How to use it

First let us look at the Basic program.
If you are building a permanent setup first rename the program in default.bas




Next go to the settings page and check the box that makes sure that default.bas is run automatically at startup.
The ESP will need a longer startup time but it automatically starts the program.

On your phone start the ESP-Voice APP

 

In the box where it says fill in your IP and the figure 192.168.1.XXX is shown replace that number by the IP number you found in your router.

If you are going to use this globally, meaning also outside of your private network, replace the IP number by xxxx.xxxx.xxxx.xxxx:8085

The xxx.xxx.xxx.xxx being the global IP adress you found on http://www.whatsmyip.org/ and the 8085 being the port number you are using on your router. This way you can use this APP anywhere in the world where you have an internet connection.

Tap the microphone icon and wait till the voice recogniser has started and speak one of the commands you entered in your Basic program and the connected led will go on or off. Neat Huh ???

Now connect the ESP's I/O ports with relays to a ledstrip like I have shown here https://lucstechblog.blogspot.nl/2016/05/esp8266-controlled-ledstrip.html or to a relay like I have shown here https://lucstechblog.blogspot.nl/2016/09/esp-relayserver.html or a string of neopixels like I described at the bottom of this story https://lucstechblog.blogspot.nl/2017/03/back-to-basic-basic-language-on-esp8266.html and start automating your home by voice commands.

That's it. A really easy way to control any devices you like by voice commands be it on your local network or from any place on the world.

Bonus

As I explained the Basic program is the same program I used for controlling the ESP by The Google Home Assistant. The consequence is therefore that you can now control the ESP in two different ways. The first is by the Home Assistant and the second is by the App on your phone. So you can control the ESP now also from rooms where there is no Home Assistant present or even from a location far way from your home.

Expansion

There is room for expansion.
Lets assume you have multiple ESP's setup for your projects. Each ESP can be equipped by the same Basic program but can have altered commands programmed. The only thing you will have to do is to alter the IP adress in the APP's field and you can adress another ESP.

If you want you can even alter the APP in such a way that you make some buttons on the screen that automatically connect to several ESP's.


Source code:

As usual you can find the source code for the APP and the APK file on github:

https://github.com/Lucvolders/ESP-Voice

So experiment and have fun.

Luc Volders

Friday, February 2, 2018

Wifi Voice Commander

For an index to all of my stories click here.

In the previous entry in this series I demonstrated how to make an App in Mit's App Inventor to control several relays attached to an ESP8266. You can re-read that story by clicking here:
https://lucstechblog.blogspot.nl/2018/01/wifi-relay.html

The reason why I showed you this method is that you do not want to have everybody in your house accessing certain electrical appliances. By using a web-page which you normally do when controlling an ESP it is easily accessible by anyone. By using an App on your phone anybody wanting to access the ESP must have the App and use the right commands to control the ESP. That gives you some extra security which can be usefull for certain purposes.

Next to that we can give the App some extra functionality: you can use spoken commands. 
Maybe you remember the Voice Command project in which I controlled an Attiny85 over bluetooth by speaking commands to my Android Phone.

Well that is exactly what we are going to do. Only we are going to do it over Wifi now. This way you can control lights/relays/pumps or whatever by speaking to them over your local network or from anywhere on the internet. Now how cool is that !!!

To achieve this you will need a program (App) on your Android phone and a program on the ESP8266. As discussed in the previous entry in this series devellopping an App for Android is easyly done with App Inventor from the MIT.

Don't worry. If this is not your thing you can download the App from the link at the bottom of this story. However then you are stuck with the commands I pre-programmed in them. You can also download the source code and alter it to your own whishes.

What ? No Iphone APP

Well lets face it, they are shamessly overpriced. I use an Alcatel Touch Pop C7 prepaid Android phone which I paid 90 Euro's for.
No way I can get an Apple Iphone for such a low price.

Next to that I do not know of any Apple App-Devellopment environment which is as easy to use as Mit's App Inventor for Android.

So till one of the readers is going to donate me a 700 Euro Apple Iphone you're out there on your own.

Lets start with making the App.

This is not going to be a tutorial on how to use App Inventor. I urge you to play around with this program and definitely have a look at all the provided examples. Ans also have a look on all the resources that can be found on the internet about App Inventor. But basically the program is so easy to use that you will be developping your own App's in no-time.

The Designer screen

Start with downloading the WifirelayVoice.aia source code to your computer. You can download it from my Github repositry with a link at the end of this story. Then point your browser to http://appinventor.mit.edu/explore/

On the right side of the screen click on the orange square that says "Create Apps"
Now you can log in with your Google credentials. If you do not have a Google account make sure you get one first.
In the projects menu on the top of your screen choose -- import project(.aia) from my computer.

 

When done your screen will look the same as above.

On the right side of the screen you will find a column with all the components that are used in your App. Clicking a component will highlight it on the example screen and in the last column on the right side of the screen you will be able to alter the settings for that component.

As you can see there are three labels that do nothing more as putting text on your screen. Then there is a textbox that allows you, when running the App, to fill in the IP adress of the ESP8266 that is going to be connected.

Next there is a large green Microphone button labeled "Voice_Input" which will activate the speechrecognizer.
And lastly there are three hidden components which are shown at the bottom of the designerscreen. The Web1 component connects the App to the internet. The TextToSpeech1 component (which was used in another project you can read by clicking here) translates written commands to spoken words. This way the app can give an audible feedback. And the SpeechRecognizer does the opposite: it translates your spoken words to plain text.




There is one small label on the screen called "Monitor".
This will display the command received from the textrecognizer on your screen. So you can see if the command you spoke was interpreted right. It is an extra check, besides the audible control, to help you analyse why a command was not executed.

By clicking on each of the components you can alter its properties like setting different backgroundcolors, altering text and fonts etc etc etc.




Just like in the previous project there is one thing I really want to Emphasize.
Look at the TextToSpeech1 component carefully. If you leave it at its default values the App will speak out any text in the language you have set as your default language in your phone. If that is English all is well. However my phone is set default to Dutch and therfore the text that will be spoken will sound very awkward. So if English is not the default language on your phone you can change the setting for the TextToSpeech1 component to English or alter the text in the program so it will speak your language.

When all is set switch, on the right (top) side of the screen, to the blocks section where the actual program will be written.

The Blocks

The program starts with initialising a global variable and setting it to empty.



 



Then the program waits till the button "Voice_Input" is clicked. If that happens the Speechrecognizer is started.

When speech has been detected the program really starts its magic.
First the label "Monitor" gets the text that is spoken. This way you can check wether the Speechrecognizer indeed recognized what you spoke.
The variable "global_input 1" gets set to the words that have been recognized.

Next a set of if-then statements check wether it is a usable command that is recognized. The commands you can use are:

- living room on
- living room off
- garage light on
- garage light of
- all lights on
- all lights off

You can alter these to your own purposes or liking. You can even alter them in your native language as long as you keep in mind that you should set the language in your phone also to your native language. You can do that in the designer screen as discussed earlier.

Let's look at one of the blocks.



the IF section tests wether the command is valid.
If that is true the Then section is performed.

In the Then section a web-command is build starting with "http://" followed by getting the IP-Adress you filled in in the main screen, and then the actual command that is getting send to the ESP. In this case that is "/sw2on".

So the complete command send over the web is: http://xxxx.xxxx.xxxx.xxxx/sw2on

And last but not least there will be an audible feed back in this case being "Garage light on"

And that's all that is to it.
Like I told you: devellopping App's with App Inventor is really easy.

You can expand this App with as many commands as you like. Just make sure you will add these command also to the ESP program which is discussed next.

The ESP8266 program.

Now the App is ready we'll have a look at the ESP side of this project. This is the same as in the previous project in this series where we used buttons to control the ESP.

This is written in C++ (generally called Arduino code) and edited and uploaded with the Arduino IDE. So I presume that you have installed the ESP8266 prerequisites in the Arduino IDE. If not do so now. The internet is full of tutorials for this so that should be no problem.

Let's have a look at the program.


 /* =================================================  
 *  Controlling NodeMCU through an Android APP  
 *  
 * Devices  
 *  ==> Relay1 represented by led 1  
 *  ==> Relay2 represented by led 2  
 *    
 * Groups   
 *  ==> All Devices ON/OFF  
 *  
 * Program by Luc Volders  
 * http://lucstechblog.blogspot.nl/  
 *=================================================*/  
 #include <ESP8266WiFi.h>  
 WiFiClient client;  
 WiFiServer server(80);  
 const char* ssid = "XXXXXXXXXX";  
 const char* password = "YYYYYYYYYY";  
 String command =""; // Command received from Android device  
 // Set led/relay Pins  
 int switch1 = 13;  
 int switch2 = 15;  
 void setup()  
 {  
  Serial.begin(115200);  
  pinMode(switch1, OUTPUT);   
  pinMode(switch2, OUTPUT);    
  digitalWrite(switch1,LOW);  
  digitalWrite(switch2,LOW);  
  connectWiFi();  
  server.begin();  
 }  
 void loop()  
 {  
   client = server.available();  
   if (!client) return;   
   command = checkClient ();  
      if (command == "sw1on") digitalWrite(switch1,HIGH);  
   else if (command == "sw1off") digitalWrite(switch1,LOW);  
   else if (command == "sw2on") digitalWrite(switch2,HIGH);     
   else if (command == "sw2off") digitalWrite(switch2,LOW);  
   else if (command == "allon")   
   {  
    digitalWrite(switch1,HIGH);  
    digitalWrite(switch2,HIGH);  
   }  
   else if (command == "alloff")   
   {  
    digitalWrite(switch1,LOW);  
    digitalWrite(switch2,LOW);  
   }  
   sendBackEcho(command); // send command echo back to android device  
   command = "";  
 }  
 /* connecting WiFi */  
 void connectWiFi()  
 {  
  Serial.println("Connecting to WIFI");  
  WiFi.begin(ssid, password);  
  while ((!(WiFi.status() == WL_CONNECTED)))  
  {  
   delay(300);  
   Serial.print("..");  
  }  
  Serial.println("");  
  Serial.println("WiFi connection established");  
  Serial.println("IP number is : ");  
  Serial.print((WiFi.localIP()));  
 }  
 /* check command received from Android Device */  
 String checkClient (void)  
 {  
  while(!client.available()) delay(1);   
  String request = client.readStringUntil('\r');  
  request.remove(0, 5);  
  request.remove(request.length()-9,9);  
  return request;  
 }  
 /* send command echo back to android device */  
 void sendBackEcho(String echo)  
 {  
  client.println("HTTP/1.1 200 OK");  
  client.println("Content-Type: text/html");  
  client.println("");  
  client.println("<!DOCTYPE HTML>");  
  client.println("<html>");  
  client.println(echo);  
  client.println("</html>");  
  client.stop();  
  delay(1);  
 }  


First replace the XXXXXX and YYYYY in the program with the name and password for your router.

As you can see I use pin 13 and 15 as the pins where the leds will be attached (which equals D7 and D8 on the NodeMCU). You can alter them for your own needs or leave them as they are. If you added more voice commands in the App you also need to assign more pins for leds.

Halfway down the program you cann see the checks for the commands sw1on and sw1off etc.
If you have altered the names for the commands in your App you will need to alter them here to. And if you have added more buttons in the App you will need to add more commands here.

The program has some serial communication which can be helpfull when debugging it. If all works flawlessly you can comment the serial communication out or delete the relevant lines.

The rest of the program takes care of receiving the commands from the App.

Upload the program to your ESP8266 and you're set to go.

The Hardware




The hardware setup is straightforward as the above breadboard setup shows you. It is again the same setup as used in the previous project in this series. For those to lazy to click to it I will give the details here again.

I have used a red led and a blue led for demonstration purposes.




The schematics are pretty straightforward to and the photo shows you my setup on a breadboard.

You can change the leds for dedicated relays like I used in the realy commander project that you can find by clicking here.

Or use a general relay controlled by a BC547 as in the project you can find by clicking here.

You can also expand it to many more leds.

Where is it

Again this should look familiar if you have read the previous story in this line-up.

In the App you need to fill in the IP adress of the ESP8266. There are 2 ways to find that IP adress.

You can use the Arduino IDE and open the serial monitor at 115200 baud. The program will show you the IP adress.

The second way is to open the web-page of your router like I did in the following picture.




The four icons on the bottom of the screen are my wired stations: a raspberry network server attached to my Epson printer, My Computer called Nescio, my Domoticz home automation system and a Buffalo Nas.
The three top Icons represent my wireless stations. The first one is an ESP that is a permanent Thermometer that serves a web-page and send info to Thingspeak (all written in ESP-Basic), my Andoid Phone and an unknown station. That last one is the interesting one.



 These are the details for the unknown wireless device and there we have the IP number we are looking for.

Real life demo




Fill in the found IP-number in the App and push the microphone button. The screen changes to a Google voice input. Say your command and see the lights go on or off.


The video shows you how the app functions although that is very obvious. You can clearly hear how the speech component works.

So what's next ??

Well in the next stories I am going to show you how easy this is accomplished with ESP-Basic instead of using C++ as in this story. 




Source code
 
As always I herebye give you the complete source code as well as the Android APK file that can be direct installed on your phone.

 https://github.com/Lucvolders/Wifi-Voice-relay
 
 

So till next time
Have fun

Luc Volders