Asking a legitimate question to a thief

It was probably September of 2005, I used to work in Manhattan on 7th Ave, between 37th and 38th street. My cousin and I went to a Deli few blocks down for lunch. We usually take away and sit in a park to eat but for some reason, we decided to sit in and eat there. The place was busy with sitting areas in two levels and we were in the second floor. I kept my bag on a chair to my right making it clearly visible to me and my cousin sat on the opposite chair facing me. While we were eating and chatting, someone lifted my bag from the chair. When I noticed that my bag was being lifted by someone, I asked the person "Why are you taking my bag?" He did not say anything, just smiled at me. I slowly took it back from his hand and placed it to the chair where it was before. Soon again I got back in the conversation with my cousin. Few minutes later, it hit me "Why was he taking my bag?" It did not occur to me that he was trying to steal my bag. For some reason I thought he needed the chair that was used to keep my bag. But then I realized few minutes later that he never really showed any interest for the chair. And he was not even there, disappeared in no time when I looked for him. He was gone! I went downstairs to report the incident to the manager but it was too late. He already took some other customers' bag from the first level and a police was in the spot to report the crime. I could not believe that I just asked a legitimate question to a thief only few minutes before the crime. What took me so long to understand his motive!