Basically, the boy (Yoshida) misunderstood that the girl (Hayama) can't handle looking directly to people's eyes. Read the first chapter to know more.