TensorFlow Multilayer perceptron
Posted: Fri Aug 04, 2017 4:23 am
www.FiveTechSoft.com
https://fivetechsoft.com/forums/
https://github.com/dmesquita/understand ... sorflow_nnFirst, create an index for each word. Then, create a matrix for each text, in which the values are 1 if a word is in the text and 0 otherwise
Code: Select all
#include "FiveWin.ch"
function Main()
local hVocabulary := hb_Hash(), hWordToIndex
local cText := "Hi from Brazil"
local cWord, aMatrix
for each cWord in TextSplit( cText )
if hb_HHasKey( hVocabulary, Lower( cWord ) )
hVocabulary[ Lower( cWord ) ] += 1
else
hVocabulary[ Lower( cWord ) ] = 1
endif
next
XBrowser( hVocabulary )
XBrowser( hWordToIndex := WordToIndex( hVocabulary ) )
aMatrix = Array( Len( hVocabulary ) )
AEval( aMatrix, { | n, i | aMatrix[ i ] := 0 } )
for each cWord in TextSplit( cText )
aMatrix[ hWordToIndex[ Lower( cWord ) ] ] += 1
next
XBrowser( aMatrix )
return nil
function TextSplit( cText )
local n, aTokens := {}
for n = 1 to NumToken( cText )
AAdd( aTokens, Token( cText,, n ) )
next
return aTokens
function WordToIndex( hVocabulary )
local hWordToIndex := hb_Hash()
local n
for n = 1 to Len( hVocabulary )
hWordToIndex[ hb_HKeyAt( hVocabulary, n ) ] = n
next
return hWordToIndex
This seems different to what I previously posted on this thread, but it seems clear that words (and text) must be turned into a bitmap so the neuronal network can process them.“It works by predicting the next letter to follow a given sequence of letters, and the predictions are determined by what it learned about language from the Friends dialogue provided,”