Artificial Intelligence: GPT with Harbour and FWH

Post Reply
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

Past July 2020, the company OpenAI released GPT3 (175 billions parameters) into private beta which has been a tremendous advance on AI natural language processing. GPT3 is the evolution of previous products and research like Microsoft Turing (17 billions parameters), NVidia Megatron (8 billions parameters), GPT2 (1.5 billions parameters), etc. In fact Google has already announced an enhanced new product named GShard that uses up to 600 billions parameters.

GPT stands for "Generative Pre Trained". This means that its work is to "generate" the next word, given some words, and that the system has been trained previously in advance.

Seven years ago Google developed and provided a "word2vec" routines that basically help to "vectorize" words. As an example for this, lets consider "colors". Colors can be represented by 3 numbers (red,green,blue) which could be represented as x,y,z coordinates (vectors) in "space". This way we can add "red" and "blue" and obtain "purple". We can see that this vector representation allows to "sum" colors, find "near" colors, "sustract" them, etc.

word2vec https://code.google.com/archive/p/word2vec/ generates these vectors from a given text, and the results are quite impressive. I have managed to build it on Windows in case you may want to test it.

Given those ideas, Rao and I decided to build a simple Harbour and FWH (as the GUI) example to check the basic concepts of a very simple GPT using Harbour. Here you have the code to build it and see how to use it to generate the next word. You can provide your own texts to it. On a next msg we will explain how it works and how you can apply it to your apps already :-)

gpt_2.prg

Code: Select all

#include "FiveWin.ch"

request DBFCDX

field WORDS

static nSize   := 3

function Main()

   local oFont, oDlg, oBrw, oGet, cScope
   local cSearch  := Space( 60 )
   local aSentences :=  { "the cat eats mouse","the mouse eats cheese",;
         "the cat eats fish","the fish eats fly", "the dog eats meat", ;
         "the dog barks at strangers", "the dog bites theives", ;
         "this is a really long long text", "this was a stage" }

   FErase( "GPT_2.dbf" )
   FErase( "GPT_2.cdx" )

   if ! File( "GPT_2.dbf" )
      DbCreate( "GPT_2.dbf", { { "ID", "+", 10, 0 }, { "WORDS", "C", 30, 0 } }, "DBFCDX" )
      USE GPT_2 VIA "DBFCDX"
      INDEX ON WORDS TAG GPT_2 CUSTOM
      USE
   endif

   USE GPT_2 VIA "DBFCDX"
   SET ORDER TO TAG GPT_2
   AEval( aSentences, { |c| RecordSentence( c ) } )

   OrdScope( 0, " " )
   OrdScope( 1, " " )
   GO TOP

   SetGetColorFocus()
   DEFINE FONT oFont NAME "TAHOMA" SIZE 0,-14
   DEFINE DIALOG oDlg SIZE 800, 350 PIXEL TRUEPIXEL FONT oFont

   @ 22,  20 SAY "Enter word(s)" SIZE 100, 20 PIXEL OF oDlg
   @ 20, 120 GET oGet VAR cSearch SIZE 660,24 PIXEL OF oDlg ;
        ON CHANGE ( If( nKey == 32, ResetBrowse( oGet:GetText(), oBrw ), nil ) ) ;
        UPDATE

   @ 60, 20 XBROWSE oBrw SIZE 760, 250 PIXEL OF oDlg ;
      DATASOURCE ReadSuggestions( "" ) ;
      COLUMNS 1, 2, 3 ;
      HEADERS "RecNo", "Suggestions", "Strength" NOBORDER

   WITH OBJECT oBrw
      :nStretchCol = 2
      :aCols[ 2 ]:bLDClickData := < ||
         oGet:cText := PadR( Trim( cSearch ) + " " + Trim( oBrw:aCols[ 2 ]:Value ), 60 )
         ResetBrowse( cSearch, oBrw )
         return nil
         >

      :CreateFromCode()
   END

   ACTIVATE DIALOG oDlg CENTERED

   OrdScope( 0, "this " ); OrdScope( 1, "this " )

   XBROWSER "GPT_2" COLUMNS "id", "words", "OrdKeyVal()", "OrdKeyCount()"

return nil

function ResetBrowse( cText, oBrw )

   local aSuggest := ReadSuggestions( cText )

   oBrw:aArrayData   := aSuggest
   oBrw:GoTop()
   oBrw:Refresh()

return nil

function ReadSuggestions( cText )

   local aRet     := {}
   local nTokens, n, nn, cKey

   cText    := Lower( AllTrim( cText ) )
   if Empty( cText )
      OrdScope( 0, " " )
      OrdScope( 1, " " )
      dbGoTop()
      DbEVal( { || AAdd( aRet, { RECNO(), WORDS, 1 } ) } )
      return aRet
   endif

   nTokens  := NumToken( cText )
   SetScope( nil )

   if !dbSeek( Token( cText ) )
      return aRet
   endif

   for n := Max( 1, nTokens - nSize + 1 ) to nTokens

      cKey  := AddTokens( cText, n, nTokens )
      nn    := NumToken( cKey )
      SetScope( cKey + " " )

      DbEVal( { || If( AScan( aRet, { |a| a[ 1 ] == RECNO() } ) == 0, AAdd( aRet, { RECNO(), WORDS, nn } ), nil ) }, ;
              { || NumToken( OrdKeyVal() ) == nn + 1 }  )

   next

return aRet

function SetScope( c )

   OrdScope( 0, c ); OrdScope( 1, c ); dbGoTop()

return nil

function RecordSentence( cText )

   local nToken, nTokens := NumToken( cText )
   local aTokens  := Array( nTokens )
   local n, cKey

   AEval( aTokens, { |u,i| aTokens[ i ] := Token( cText, nil, i ) } )

   SET ORDER TO TAG GPT_2

   if !dbSeek( " " + aTokens[ 1 ] )
      dbAppend()
      WORDS   := aTokens[ 1 ]
      OrdKeyAdd( "GPT_2", nil, " " + Lower( aTokens[ 1 ] ) )
   endif

   for nToken := 1 to Len( aTokens ) - 1
      cKey  := Lower( aTokens[ nToken ] + " " + aTokens[ nToken + 1 ] )
      if !dbSeek( cKey )
         dbAppend()
         WORDS   := aTokens[ nToken + 1 ]
         OrdKeyAdd( "GPT_2", nil, cKey )
         if nToken > 1
            for n := nToken -1 to Max( 1, nToken - nSize ) step - 1
               cKey  := Lower( AddTokens( cText, n, nToken + 1 ) )
               OrdKeyAdd( "GPT_2", nil, cKey )
            next
         endif
      endif
   next

return nil

function LastTokens( cText, nCount )

   local nTokens  := NumToken( cText )
   local n
   local nFrom    := Max( nTokens - nCount + 1, 1 )
   local cRet     := Token( cText, , nFrom )

   for n := nFrom + 1 to nTokens
      cRet  += " " + Token( cText, , n )
   next

return cRet

function AddTokens( cText, nFrom, nLast )

   local n, cRet

   DEFAULT nLast  := NumToken( cText )
   nFrom    := Max( 1, nFrom )
   cRet     := Token( cText, , nFrom )

   for n := nFrom + 1 to nLast
      cRet  += " " + Token( cText, , n )
   next

return cRet
 
Image
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

Though OpenAI GPT3 is in closed private beta, there is an easy to way to start using it! :-)
Go to https://play.aidungeon.io/ select option 6 and use these chat models with the historical characters of your choice.
Be aware that all these answers are fiction so they may be unaccurate or totally false. Enjoy it!

These are some examples of chats with historical characters using OpenAI GPT3:

An interview with Steve Jobs: (some of my questions were written by the AI also!)
My name is Antonio and I went to California to meet Steve Jobs. Antonio: Hi Steve, what a great honor to meet you. Steve: You are welcome. Antonio: what inspired you to build the iphone ?
Steve: The iPhone was an idea that came to me while I was in Hawaii.
Antonio: what phone did you use by that time ? Steve: I was using a Motorola V171. Antonio: how did you choose that phone and how many tests did you make before the launch of the iphone? Steve: My team and I picked the phone for several reasons.
Antonio: Could you explain those reasons ? Steve: Yes, first we chose a CDMA carrier so that it would work in all of the US. Second, we wanted a phone with a good camera.
Antonio: was it clear to you that it was going to be a true computer ? Steve: It was not a computer per say, but we knew that it could do many of the things that computers can do.
Antonio: do you know how long did it take you to build it?
Steve: The phone took around 3 months to build.
Antonio: what do you think about the reactions of the phone?
Steve: I think it's the greatest invention since the computer.
Antonio: is it true that you were thinking about calling it "the iPod phone"?
Steve: It was discussed, but the name "iPhone" got more votes.
Antonio: is it true that you were offered a lot of money by various companies to stop developing it?
Steve: I was offered several million dollars to abandon the project, but I knew it was a good idea and we would make it successful.
Antonio: what about Microsoft ? what was its reaction ? Steve: The first reaction of most people was WOW, then some were skeptical, some wanted to know more, and others tried to steal the idea.
Antonio: Anyhow the first iphone versions lacked enough quality. How do you explain the overall positive reaction to it ? Steve: I don't know, I think it's a great phone and people were just surprised that it was this good.
Antonio: What about the "antennagate" controversy?
Steve: That was a another controversy that didn't get as much attention as it should have. The phone has great reception, I'd change the design though if I could do it all over again.
Antonio: Did you manage an anternative design ? Steve: Yes, we tried many designs before arriving at the current one. The search was not easy.
Antonio: what was the most difficult part about the process?
Steve: The most difficult part of the process was making sure that it could easily be mass-produced at a cost that was reasonable to consumers.
Antonio: How many iphones were made in the first days?
Steve: We made around 10,000 iphones in the first week.
Antonio: what happened to the phones after that? Steve: We continued to make them as fast as we could.
Antonio: Surely we became very satisfied about it. Steve: I'm glad to see that, because Apple is working on new features for the next version.
Antonio: Could you tell us more about the features?
Steve: Sure, there are a few new features that we are working on.
Antonio: Do you think it is going to be a big success this time?
Steve: I think this is going to be a big success, Apple has a reputation to uphold and the iphone is going to do that.
Antonio: Have you ever experienced a period of great personal pressure?
Steve: When I first started working on the project, there were many times when I didn't think we were going to succeed, but the great thing is that my team and I believed in the idea so much that we persevered
Antonio: are there any flaws still in its design ? Steve: The phone has good design, but we are always trying to make it better.
Antonio: Could you tell us about some of the improvements you are going to make in the future?
Steve: No, I don't think I should, but we are always trying to make the product better.
Antonio: one more question, do you think it is going to be popular among the kids?
Steve: I don't think so, kids usually don't want to be bothered with making calls and using the internet.
Antonio: It's funny because it's true.
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Silvio.Falconi
Posts: 4956
Joined: Thu Oct 18, 2012 7:17 pm

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Silvio.Falconi »

i tried also
You are Antonio Linares, a person trying to stay sane during a global quarantine. You have a ton of free time and a rubik's cube that you will never finish. You are bored out of your mind and have run out of things to do. But then you notice in the corner of your room something strange. There is a small box with a glowing green button on it. You have seen this box before, in the corner of your parent's room. You think to yourself, "This is so cool!"

> You press the green button.
The button makes a flashing green light appear on the box. Soon, there is a low, rumbling noise coming from all sides of the room.
> You say "I am hurt"
The lights begin to flicker on and off.
"I am hurt," you say.
Suddenly, something appears from the box.

> You say "the Silvio Falconi Photo"
A dark figure appears and hovers in the air in front of you.
"The Silvio Falconi photo!" you say.
"Noooo!"

> You say "Perhaps is NagesWararo, my indian friend"
A second figure appears and hovers in the air in front of you.
"Maybe it's NagesWararo, my indian friend," you say.
"I Am Not NagesWararo!

> You say "Who are you?"
"I am the spirit of Christmas Future," the spirit says in a thick Indian accent.

> You say "today is not Xmax , you must come back until 3 month "
"No, no, no, no! Nobody knows I am here! Not the humans, not the ghosts, not the weird creatures, no!"
The spirit floats around your room, becoming more and more agitated.

I open the windows !" he says, "and the humans will catch me! I must leave before they find me!"
Suddenly, the lights flicker off.
"I cannot fly in the dark! Humans are coming!


aahahahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahhahah
I use : FiveWin for Harbour August 2020 (Revision) - Harbour 3.2.0dev (r1712141320) - Bcc7.30 - xMate ver. 1.15.3 - PellesC
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

great! ;-D
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
Posts: 37481
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain
Contact:

Re: Artificial Intelligence: GPT with Harbour and FWH

Post by Antonio Linares »

Another site using GPT2

https://transformer.huggingface.co/
regards, saludos

Antonio Linares
www.fivetechsoft.com
Post Reply